OpenCL kernel headers not found (macOS)

[cintainfinita]

MacOS BigSur, AMD Radeon Pro 5700 XT 16 GB

% ./john --list=build-info Version: 1.9.0-jumbo-1+bleeding-b49c763c4 2021-06-24 21:32:53 +0200 Build: darwin20.5.0 64-bit x86_64 AVX2 AC SIMD: AVX2, interleaving: MD4:4 MD5:5 SHA1:2 SHA256:1 SHA512:1 CPU tests: AVX2 $JOHN is ./ Format interface version: 14 Max. number of reported tunable costs: 4 Rec file version: REC4 Charset file version: CHR3 CHARSET_MIN: 1 (0x01) CHARSET_MAX: 255 (0xff) CHARSET_LENGTH: 24 SALT_HASH_SIZE: 1048576 SINGLE_IDX_MAX: 2147483648 SINGLE_BUF_MAX: 4294967295 Effective limit: Number of salts vs. SingleMaxBufferSize Max. Markov mode level: 400 Max. Markov mode password length: 30 clang version: 12.0.5 (clang-1205.0.22.11) (gcc 4.2.1 compatibility) OpenCL headers version: 1.2 Crypto library: OpenSSL OpenSSL library version: 0101010bf OpenSSL 1.1.1k 25 Mar 2021 File locking: fcntl() fseek(): fseek ftell(): ftell fopen(): fopen memmem(): System's times(2) sysconf(_SC_CLK_TCK) is 100 Using times(2) for timers, resolution 10 ms HR timer: mach_absolute_time(), latency 33 ns Total physical host memory: 72 GiB Available physical host memory: 56299 MiB Terminal locale string: UTF-8 Parsed terminal locale: UTF-8

Path error:

% ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I opencl -cl-mad-enable -DOS_X -DGPU__ -DDEVICE_INFO=522 -DSIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ./opencl/pbkdf1_hmac_sha1_kernel.cl Build log: :47:10: fatal error: 'opencl_device_info.h' file not found

include "opencl_device_info.h"

     ^

Error building kernel ./opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522 0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1292 - clBuildProgram

I tried running john from a different path, no luck.

I specified the full path to the include instruction but still the file wasn't found.

[claudioandre-br]

Off the top of my head, to get around this you need to run john (at least once) from inside the opencl/ folder.

cd opencl
../john --test --format=opencl

Well, maybe you need to find the "right location" (if the 'opencl' folder does not fix it), as stated here https://www.openwall.com/lists/john-users/2014/08/03/2 (notice, this is the old headers location).

"OS X also has a problem with run-time compile of kernels that include header files. A workaround is to cd to the src directory and run each OpenCL format once. After that, the kernel binary is cached so you can move away from the src directory."

[solardiz]

So the OpenCL compiler somehow ignored -I opencl that was passed to it. I wonder why this issue keeps occasionally reoccurring on various systems. Is there anything we're doing wrong (I don't think so)? Is there anything we can do better (e.g., maybe -I . along with opencl/ in paths inside the #include directives would have worked better across systems)?

@magnumripper Don't you use a similar Mac system now, yet aren't running into this issue?

[cintainfinita]

@claudioandre-br tried that (per that post, it mentions the "kernels" directory that is not there anymore, replaced by opencl, so I already did that) with no luck.

The odd part is that I modified the #include's to match my exact location of the .h files (full path) and it still says the file is not found.

This is a very old problem, I tried all the tricks from part workaround and none of them worked. Also, the kernels directory was moved to the opencl directory.

If I can help compiling and test any changes, count on me.

Thanks!

[magnumripper]

On 2021-06-25 14:39, Solar Designer wrote:

So the OpenCL compiler somehow ignored -I opencl that was passed to it. I wonder why this issue keeps occasionally reoccurring on various systems. Is there anything we're doing wrong (I don't think so)? Is there anything we can do better (e.g., maybe -I . along with opencl/ in paths inside the #include directives would have worked better across systems)?

@magnumripper Don't you use a similar Mac system now, yet aren't running into this issue?

I haven't heard of that issue for years. If I ever did see it myself I'm sure I committed workarounds that should still be in place.

magnum

[magnumripper]

What if you move all files in run/opencl to just run? That'd be ugly but it would be interesting to see if it matters.

[cintainfinita]

Moved everything to the run/opencl folder and made the file structure also:

% pwd
[REDACTED]/Tools/john/run/opencl
% ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Can't read source kernel: No such file or directory

That's the output.

[claudioandre-br]

What if you move all files in run/opencl to just run? That'd be ugly but it would be interesting to see if it matters.

Copying (not moving). And enable OpenCL verbose (the way it works in MacOS):

• To check if the Mac is compiling the kernel in some weird folder like /temp/cl88ddSdoui (it used to, right?)
• Well, hope we can see something new in the outputs.

[cintainfinita]

Hi @claudioandre-br and @magnumripper !

I don't really know how to enable verbosity for OpenCL in MacOS. I Googled it but didn't find anything useful. If you can help on that, I can try it.

I compiled a fresh copy and now instead of moving I copied the needed files to the run/opencl folder. Same error.

Running John binary from different locations:

moebius@kandinsky opencl % ../john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ../opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: <program source>:47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
         ^

Error building kernel ../opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1292 - clBuildProgram

moebius@kandinsky opencl % ./john --test --format=opencl 
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Can't read source kernel: No such file or directory

Thanks!

[magnumripper]

What if you move all files in run/opencl to just run? That'd be ugly but it would be interesting to see if it matters.

Copying (not moving). And enable OpenCL verbose (the way it works in MacOS):

Right, also move the headers to run, not the CPU binaries to opencl. From a normal tree it'd be like cd run; cp opencl/*.h .. However, it wont solve the below:

• To check if the Mac is compiling the kernel in some weird folder like /temp/cl88ddSdoui (it used to, right?)

Ah yes this sounds familiar. I fixed it at some point by starting to pass -I /full/path/to/john/opencl to the kernel runtime build instead of just eg. -I opencl but apparently that doesn't seem to kick in according to OP. Did we break it recently?

I wonder why I'm not seeing those problems at all? I run Big Sur on intel and M1 macbooks.

[magnumripper]

I also get just -I opencl but it does work for me.

[magnumripper]

Ah yes this sounds familiar. I fixed it at some point by starting to pass -I /full/path/to/john/opencl to the kernel runtime build instead of just eg. -I opencl but apparently that doesn't seem to kick in according to OP. Did we break it recently?

Looks like @solardiz broke it in 753dd9f102.

[magnumripper]

We could want to revert 753dd9f10 but change the inital ifdefs to eg, #if __APPLE__

-#if !defined(__CYGWIN__) && !defined(__MINGW32__)
+#if __APPLE__
+// If true, use realpath(3) for translating eg. "-I./opencl" into an absolute
+// path before submitting as JIT compile option to OpenCL.
+#define I_REALPATH 1
+#endif

Why am I not affected by these problems though? I'm not sure I ever was, using Macs since 2012 or so.

[solardiz]

Would we also avoid the issue by chdir to opencl/ and passing -I .? If so, I think that's the way to go, and we could do it unconditionally on all OSes.

[cintainfinita]

@magnumripper I've been using John in Macs since 2008 and this is also the first time. I moved from an old iMac (2010, ATI reader Graphics Card) to this new one, and I found this error when doing the compilation process and some benchmarks. First time too, I can state that OpenCL was running OK on the old iMac. Maybe the old MacOS is doing something different?

Let me know if I can help testing any changes on the code.

Thanks.

[magnumripper]

Would we also avoid the issue by chdir to opencl/ and passing -I .?

No - as Claudio reminded us, some versions of MacOS and/or drivers will copy the kernel file (supplied as a char array IIRC) to some temporary directory. I can't see any way to fix the issue except supplying a full path to our opencl directory. Well, unless we do a cpp stage ourselves... that'd be a nice can of worms, LOL...

I assume this only happens with particular combos of devices and driver versions and that would explain why I never had the problem myself (or TBH maybe I did have it at some point long ago and don't even remember).

[claudioandre-br]

We could want to revert 753dd9f but change the inital ifdefs to eg, #if __APPLE__

-#if !defined(__CYGWIN__) && !defined(__MINGW32__)
+#if __APPLE__
+// If true, use realpath(3) for translating eg. "-I./opencl" into an absolute
+// path before submitting as JIT compile option to OpenCL.
+#define I_REALPATH 1
+#endif

Why am I not affected by these problems though? I'm not sure I ever was, using Macs since 2012 or so.

I vote to apply (something like) this:

• Nobody is going to create and test a general solution for Windows and Mac.

[cintainfinita]

@claudioandre-br if you patch something, I can pull the code and test it in my MacOS. I can try an older version too... thanks!

[magnumripper]

@cintainfinita try using latest & greatest code, then just git revert 753dd9f. Then build and test and please report back.

[cintainfinita]

@magnumripper

% git revert 753dd9f
Auto-merging src/opencl_common.c
[bleeding-jumbo 48a953279] Revert "OpenCL: Build kernels having chdir'ed to John's home directory"
 Committer: moebius <moebius@kandinsky.local>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

    git config --global --edit

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 1 file changed, 109 insertions(+), 45 deletions(-)

No luck, same error:

moebius@kandinsky opencl % ../john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I /Users/moebius/Documents/Tools/john/run/opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ../opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: <program source>:47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
         ^

Error building kernel ../opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1356 - clBuildProgram
moebius@kandinsky opencl % cd ..      
moebius@kandinsky run % ./john --test --format=opencl 
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I /Users/moebius/Documents/Tools/john/run/opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ./opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: <program source>:47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
         ^

Error building kernel ./opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1356 - clBuildProgram

[magnumripper]

This seems to be a different problem than the "temporary directory" one. I have no idea what is happening there. Maybe try building latest hashcat and see if you get the same problem with that?

[magnumripper]

Would we also avoid the issue by chdir to opencl/ and passing -I .? If so, I think that's the way to go, and we could do it unconditionally on all OSes.

Maybe we should actually try this here.

[cintainfinita]

Hi @magnumripper.

Look!

% ./hashcat -D 1 -b   
hashcat (v6.2.2-70-g5bea13200) starting in benchmark mode

Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.

OpenCL API (OpenCL 1.2 (May  8 2021 03:14:28)) - Platform #1 [Apple]
====================================================================
* Device #1: Intel(R) Core(TM) i9-10910 CPU @ 3.60GHz, 73664/73728 MB (18432 MB allocatable), 20MCU
* Device #2: AMD Radeon Pro 5700 XT Compute Engine, skipped

Benchmark relevant options:
===========================
* --opencl-device-types=1
* --optimized-kernel-enable

Hashmode: 0 - MD5

clCompileProgram(): CL_BUILD_PROGRAM_FAILURE

<program source>:7:10: fatal error: 'inc_vendor.h' file not found
#include "inc_vendor.h"
         ^

* Device #1: Kernel /Users/moebius/Documents/Tools/hashcat/OpenCL/shared.cl build failed.

Started: Tue Jun 29 09:55:04 2021
Stopped: Tue Jun 29 09:55:04 2021

Looks like hashcat is "affected" too. I will see if I can find anything useful here for John.

[cintainfinita]

Update on hashcat:

1)I saw someone stating that installing hashcat using brew solved the problem. And it worked. hashcat worked.

2) sudo make install on the hashcat code. run it outside the hashcat's directory. worked!

[magnumripper]

installing hashcat using brew solved the problem

That's odd, I can't see how/why it would matter. How did you build John? If you didn't use brew, maybe try the macOS instructions at bottom of doc/INSTALL:

  1. Install Xcode (free in App Store).
  2. Install Xcode's command-line tools:
        xcode-select --install
     You'll need to re-run that command after any little update of macOS,
     because the command-line tools silently disappeared and you'll get a silly
     "error: C compiler cannot create executables" when trying to build John.
  3. Install Homebrew, by following the instructions given here:
        https://brew.sh
  4. Install Homebrew's gcc and OpenSSL:
        brew install gcc openssl
  5. Consider adding homebrew directories such as /usr/local/bin and/or
     /opt/homebrew/bin to your $PATH.
  6. Configure, possibly adding a CC option for pointing to a specific gcc and
     using whatever LDFLAGS and CPPFLAGS was recommended when you installed
     Homebrew's OpenSSL:
        ./configure CC="/opt/homebrew/bin/gcc-10" \
          LDFLAGS="-L/opt/homebrew/opt/openssl/lib" \
          CPPFLAGS="-I/opt/homebrew/opt/openssl/include"
  7. Clean old files and make:
        make -s clean && make -sj4

If this does solve the problem, I'm really stumped as of why!

[magnumripper]

From what I understand in https://github.com/hashcat/hashcat/issues/2270, the key for hashcat isn't really using brew, but running "make install". That makes even less sense.

[magnumripper]

We need to test this before digging more into this - @cintainfinita try applying this patch to a clean tree (EDIT: or pull #4762):

diff --git a/src/opencl_common.c b/src/opencl_common.c
index 17ff13871..42d8f090a 100644
--- a/src/opencl_common.c
+++ b/src/opencl_common.c
@@ -1146,7 +1146,7 @@ static char *get_build_opts(int sequential_id, const char *opts)
            global_opts = OPENCLBUILDOPTIONS;

    snprintf(build_opts, LINE_BUFFER_SIZE,
-            "-I opencl %s %s%s%s%s%d %s%d %s -D_OPENCL_COMPILER %s",
+            "%s %s%s%s%s%d %s%d %s -D_OPENCL_COMPILER %s",
            global_opts,
            get_platform_vendor_id(get_platform_id(sequential_id)) ==
             PLATFORM_MESA ? "-D__MESA__ " :
@@ -1228,12 +1228,7 @@ void opencl_build(int sequential_id, const char *opts, int save, const char *fil
 #endif /* HAVE_MPI */

 /*
- * Build kernels having temporarily chdir'ed to John's home directory.
- *
- * This lets us use simply "-I opencl" instead of having to resolve a pathname,
- * which might contain spaces (which we'd have to quote) and was problematic on
- * Cygwin when run from Windows PowerShell (apparently, with Cygwin resolving
- * pathnames differently than the OpenCL backend would for the includes).
+ * Build kernels having temporarily chdir'ed to John's opencl directory.
  *
  * Saving and restoring of the current directory here is incompatible with
  * concurrent kernel builds by multiple threads, like we'd do with the
@@ -1247,18 +1242,18 @@ void opencl_build(int sequential_id, const char *opts, int save, const char *fil
    int old_cwd_fd = -1;
    char old_cwd[PATH_BUFFER_SIZE];
    old_cwd[0] = 0;
-   char *john_home = (char *)path_expand_safe("$JOHN/");
-   if (john_home[0] && strcmp(john_home, "./")) {
+   char *john_opencl = (char *)path_expand_safe("$JOHN/opencl");
+   if (john_opencl[0] && strcmp(john_opencl, "./opencl")) {
        old_cwd_fd = open(".", O_RDONLY);
        if (!getcwd(old_cwd, sizeof(old_cwd))) {
            old_cwd[0] = 0;
            if (old_cwd_fd < 0)
                fprintf(stderr, "Warning: Cannot save current directory: %s\n", strerror(errno));
        }
-       if (chdir(john_home))
-           pexit("chdir: %s", john_home);
+       if (chdir(john_opencl))
+           pexit("chdir: %s", john_opencl);
    }
-   MEM_FREE(john_home);
+   MEM_FREE(john_opencl);
    build_code = clBuildProgram(*program, 0, NULL, build_opts, NULL, NULL);
    if ((old_cwd_fd >= 0 || old_cwd[0]) && /* We'll only have errno when we attempt a *chdir() here */
        (old_cwd_fd < 0 || fchdir(old_cwd_fd)) && (!old_cwd[0] || chdir(old_cwd)))

It works just fine on my macBook (but that one didn't have problems before either). I opted to not include -I . on line 1149 because it really shouldn't be needed, but in case you still have problems with the above patch, try re-adding -I . to that line where it previously said -I opencl.

[magnumripper]

@solardiz on a side note I wonder whether using #include <opencl_device_info.h> would work better than #include "opencl_device_info.h" for the original code without the above patch... I'm thinking -I is normally used for finding <file.h> whereas "file.h" would be looked for in current directory, right?

Anyway, in case the above patch happens to work it's probably better (safer on other platforms).

EDIT: I just tried this for phun:

#include "stdio.h"
int main(int argc, char **argv)
{
    while (argc--)
        printf("%s ", *argv++);
    printf("\n");
}

All compilers I tried finds stdio.h even though listed with quotes instead of <>. Some googling indicates this is normal manner - quotes just tries current dir before the others.

[solardiz]

I'm thinking -I is normally used for finding <file.h> whereas "file.h" would be looked for in current directory, right?

I think you're right. We can try that.

[cintainfinita]

Hi all! thanks for all your efforts!

1) Tried the patch without the -I. No luck, .h still not found. Added the -I. Same error.

2) I modified the .cl, changing "test.h" with . Same result, file not found.

To compile john, I just used ./configure and then the suggested make commands. I have openssl headers, etc.

Maybe something related to GCC?, here's my output:

% gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

So, I brew installed GCC-10 and point the configure process to the new one: % ./configure CC="/usr/local/Cellar/gcc@10/10.3.0/bin/gcc-10"

% ./john --test --format=opencl                              
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ./opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: <program source>:47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
         ^

Error building kernel ./opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1287 - clBuildProgram

Sorry, can't make it work. Let me know. Thanks!

[magnumripper]

It shouldn't be related at all to the host code's compiler or using brew or not. The thing we see failing is the OpenCL subsystem trying to build the kernel - and it's definitely using Apple's framework which we have little control over.

I see no way around this other than either just declaring any system behaving this way b0rken (as in not our problem) - or actually try a cpp pass ourselves. The latter would be pretty silly...

[magnumripper]

BTW I've been missing strace on macOS. Maybe there is some equivalent thing one could use to get a clue.

EDIT: there is dtrace but it's so limited by SIP it's a real hassle to use (not to mention understanding dtrace in the first place)

[solardiz]

Update on hashcat:

1)I saw someone stating that installing hashcat using brew solved the problem. And it worked. hashcat worked.

2. sudo make install on the hashcat code. run it outside the hashcat's directory. worked!

In those tests, did you by any chance run hashcat as root? Maybe running it as root is what makes the difference?

[solardiz]

2\. I modified the .cl, changing "test.h" with <test.h>. Same result, file not found.

Did you test this with or/and without -I?

[cintainfinita]

With and without! both cases.

[magnumripper]

Just to verify that things can work at all, try this (from a pristine tree):

Edit bf_kernel.cl. Drop the line #include "opencl_device_info.h" line from it and instead insert the contents of opencl_device_info.h at that place.

Then get back to run directory and ./john -test=0 -format=bcrypt-opencl

I guess that will work but if it doesn't, our problem may not be what it seems.

[cintainfinita]

@magnumripper haha! you rock! That worked!!!!

% ./john -test=0 -format=bcrypt-opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
LWS=8 GWS=4096 Testing: bcrypt-opencl ("$2a$05", 32 iterations) [Blowfish OpenCL]... PASS

Ok, it is not the perfect way but this will work everywhere. This is a workaround and I will have to do this for all my files or maybe you can push an update on the main repo so a clean install will always work? What are your thoughts? Thanks!

[magnumripper]

@cintainfinita what I wrote about us doing a cpp pass right before compiling the kernel is more or less the same thing, but using some external tool (so not trivially portable). You simply did manually what the run-time OpenCL compiler failed to do.

As a one-off workaround, I could probably come up with a one-liner bash ninja trick for you, which would do that automagically for every kernel in one fell swoop, and we could even include that in a make target but I'd rather understand what the heck is going on in your gear and how to work around it properly. Let me sleep on this a little more!

[cintainfinita]

@magnumripper Awesome! Yes, I now that the compiler works that way... hahaha.. great.. maybe that bash could be in the "tools" section or something related and some instructions in the README file... I can help testing more and finding the key problem, no worries. In the meantime, I know now (and everybody else with this issue) how to fix this if the format with opencl is needed.

Let me know what else we can try!

Thanks! Sleep well.

[magnumripper]

Can you try a make uninstall with hashcat, and see if the problem comes back then?

[magnumripper]

Had a look at hashcat. They use quotes and no path component for all headers in their kernels, just like us. The headers and kernels reside in OpenCL/, much like ours.

#include "inc_vendor.h"

At some point, they did a chdir to the OpenCL directory but it's currently disabled: Their current code doesn't chdir at all anymore (except for when restoring but that's unrelated). Instead they use -I OpenCL -I "/full/path/to/OpenCL" for the run-time build. The full path is always quoted and the relative path -I OpenCL is a static string, sent even when not correct.

I can't see any reason why it would work better (for some) after make install and executed with no path.

@magnumripper

% git revert 753dd9f
Auto-merging src/opencl_common.c
(...)
 1 file changed, 109 insertions(+), 45 deletions(-)

No luck, same error:

@cintainfinita are you sure you re-built after reverting that commit, before testing the result? I know anyone (as in me too) can have such moments...

[cintainfinita]

hey @magnumripper ! yup, did that again, revert, configure, make clean, make.

The weirdest thing is related to hashcat and brew. If you install it with brew it will work well too.

As of now, the only solution is to do the compiler's job haha... manually including the ".h" within each ".cl". Hope this helps, at least, any people reaching this conversation and looking for a quick workaround to crack something.

I'm here to help testing any potential solution if you need. Thanks.

[magnumripper]

Here's a crazy one-liner for you, run it in the opencl directory:

while grep -q ^#include *.cl; do for i in `grep -l ^#include *.cl`; do mv $i temp.cl; for j in `grep ^#include temp.cl | cut -d\" -f2`; do cat $j >> $i ; done ; grep -v ^#include temp.cl >> $i ; done ; rm temp.cl ; done

[cintainfinita]

Thanks! Did that. Just in case, I recompiled (I don't know if this step is a must, don't think so, but just in case). First test case ran well, but I believe I have some missing dependencies or something else wrong now.... Gggrrr.. let me know if this is related to this particular case or if I should Google that a bit... Thanks!

% ../john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... LWS=64 GWS=163840 (2560 blocks) DONE
Speed for cost 1 (iteration count) of 20000
Raw:    153121 c/s real, 16384K c/s virtual

Benchmarking: KeePass-opencl [SHA256 AES/Twofish/ChaCha OpenCL]... Options used: -I /Users/moebius/Documents/Tools/john/run/opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DPLAINTEXT_LENGTH=124 -DHASH_LOOPS=100 -DMAX_CONT_SIZE=16777216 ../opencl/keepass_kernel.cl
Build log: <program source>:2406:33: error: unknown type name 'AES_KEY_TYPE'
inline void AES_set_encrypt_key(AES_KEY_TYPE void *_userKey,
                                ^
<program source>:2406:46: error: expected ')'
inline void AES_set_encrypt_key(AES_KEY_TYPE void *_userKey,
                                             ^
<program source>:2406:32: note: to match this '('
inline void AES_set_encrypt_key(AES_KEY_TYPE void *_userKey,
                               ^
[... much more, snipped by magnum ...]

Error building kernel ../opencl/keepass_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1356 - clBuildProgram
moebius@kandinsky opencl % 

[magnumripper]

Oh, OK. I think some of the things came out in the wrong order :-(

I'll come up with a better way of doing it later, possibly actually using CPP.

[cintainfinita]

haha! no worries. yeap, maybe it was too crazy for this delicate iMac hahahaha... I'll be around if you need more testing later. Thanks!

[claudioandre-br]

are you sure you re-built after reverting that commit, before testing the result? I know anyone (as in me too) can have such moments...

Does Mac has a trick to remove its "OpenCL compiled binaries"? Are they .bin files inside opencl? This is something to watch out for when testing.

IMO, or it is something bad inside Apple drivers or in testing procedure. Seems clear JtR is doing the right thing (TM).

PS: Ok, maybe we need the REALPATH patch back.

[magnumripper]

Does Mac has a trick to remove its "OpenCL compiled binaries"? Are they .bin files inside opencl? This is something to watch out for when testing.

Yes, rm opencl/*.bin or make kernel-cache-clean, but there will be nothing cached for kernels that never built successfully.

IMO, or it is something bad inside Apple drivers or in testing procedure. Seems clear JtR is doing the right thing (TM).

PS: Ok, maybe we need the REALPATH patch back.

I think we should consider chdir to opencl instead of run, even though it didn't help here.

[magnumripper]

Oh, OK. I think some of the things came out in the wrong order :-(

Here's next hack. CD to the opencl directory and paste this to the terminal:

cat <<'EOF' >bugpp.pl
#!/usr/bin/perl -w
use strict;
sub process_file {
    my $file = shift @_;
    open my $fh, "<", $file or die "$file: $!";
    binmode $fh, ":raw";
    while (<$fh>) {
        if (/^\s*#\s*include\s+"([^"]+)"/) {
            process_file($1);
        } else {
            print;
        }
    }
    close($fh);
}
process_file($ARGV[0]);
EOF
for i in *.cl; do perl bugpp.pl $i >x && mv x $i ; done

[cintainfinita]

I removed everything just in case. Cloned a fresh copy of JtR. Ch'ddir to run/opencl and ran the hack. Compiled.

moebius@kandinsky run % ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... LWS=64 GWS=163840 (2560 blocks) DONE
Speed for cost 1 (iteration count) of 20000
Raw:    152409 c/s real, 16384K c/s virtual

Benchmarking: KeePass-opencl [SHA256 AES/Twofish/ChaCha OpenCL]... LWS=32 GWS=327680 (10240 blocks) DONE
Speed for cost 1 (iteration count) of 50000 and 6000, cost 2 (version) of 1 and 2, cost 3 (algorithm [0=AES 1=TwoFish 2=ChaCha]) of 0
Raw:    35772 c/s real, 5957K c/s virtual

Benchmarking: oldoffice-opencl, MS Office <= 2003 [MD5/SHA1 RC4 OpenCL/mask accel]... |

Worked like a charm! For all formats.

This little thread and hack will save the time of some people, I'm sure. Good workaround, not the best solution and we still don't know what is going on, but anyway, it does the job...

If you guys still want to dig further, count on me.

Thanks!

[magnumripper]

@solardiz before release, perhaps we should document this problem, include that little perl hack and add a make target for running it if needed, eg. make kernel-pp.