search

openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.01k stars 2.07k forks source link

Mask is unable to properly feed forked processes #3714

Open dosoos opened 5 years ago

dosoos commented 5 years ago

~# john John the Ripper 1.8.0.16-jumbo-1-bleeding-4840892d68 2019-03-26 21:33:03 -0300 OMP [linux-gnu 64-bit x86_64 AVX2 AC] Copyright (c) 1996-2019 by Solar Designer and others Homepage: http://www.openwall.com/john/

command: john --format=oldoffice-opencl --fork=8 --mask=?d?d?d?d?d?d?d?d xxoo.xls.hash

this version crack this hash: xxooxxoo.file:$oldoffice$0b7663a051324739135c06ad1362bff35fb5a3178a5e8cfc7c19ebb9ff85f8ede*6b59b44de0ca2d58ccfbfa0138bebe50:::::xxoo.xls

response: Using default input encoding: UTF-8 Loaded 1 password hash (oldoffice-opencl, MS Office <= 2003 [MD5/SHA1 RC4 OpenCL]) Cost 1 (hash type) is 0 for all loaded hashes Node numbers 1-8 of 8 (fork) 1: Local worksize (LWS) 64, global worksize (GWS) 40960 (640 blocks) Press 'q' or Ctrl-C to abort, almost any other key for status 5: Local worksize (LWS) 64, global worksize (GWS) 81920 (1280 blocks) 1: Warning: Only 2120 candidates left, minimum 40960 needed for performance. 1: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 6: Local worksize (LWS) 64, global worksize (GWS) 40960 (640 blocks) 8: Local worksize (LWS) 64, global worksize (GWS) 40960 (640 blocks) 3: Local worksize (LWS) 64, global worksize (GWS) 40960 (640 blocks) 4: Local worksize (LWS) 64, global worksize (GWS) 40960 (640 blocks) 7: Local worksize (LWS) 64, global worksize (GWS) 40960 (640 blocks) root@gpu-server4:/usr/local/JohnTheRipper-bleeding-jumbo/run# 5: Warning: Only 43080 candidates left, minimum 81920 needed for performance. 5: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 6: Warning: Only 2120 candidates left, minimum 40960 needed for performance. 6: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 2: Local worksize (LWS) 64, global worksize (GWS) 40960 (640 blocks) 8: Warning: Only 2120 candidates left, minimum 40960 needed for performance. 3: Warning: Only 2120 candidates left, minimum 40960 needed for performance. 4: Warning: Only 2120 candidates left, minimum 40960 needed for performance. 7: Warning: Only 2120 candidates left, minimum 40960 needed for performance. 8: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 3: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 4: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 7: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt

claudioandre-br commented 5 years ago

What we know:

CL_OUT_OF_RESOURCES if there is a failure to allocate resources required by the OpenCL implementation on the device.

Happens:

It is unlikely the code has a pointer issue, since I can run basically the same here:

That being said, you should try to narrow the cause down. E.g., run:

solardiz commented 5 years ago

@dosoos How many GPUs do you have in that machine? If it's substantially fewer than 8, then you shouldn't have used --fork=8. Perhaps you're simply overloading your GPU(s)/driver(s) by running too many concurrent jobs on them with that fork. Please confirm that this was the user error you made, so that we can close this issue.

For further occasions, if any, please use the john-users mailing list for user support. Our use of GitHub issues is for tracking bugs and planned development, not for user support. Thanks.

solardiz commented 5 years ago

Hmm, if #3693 is from the same machine (BTW, that one was even more obviously a user support request), then maybe you actually have 8 GPUs there and the --fork=8 was OK.

solardiz commented 5 years ago

I assume device names are not printed because this build is missing commit 666f54f8f15480b9dac9953bf5d4007b887560f0. It could be helpful to reproduce the issue with the latest code, and post the corresponding output including device names.

dosoos commented 5 years ago

my machine have 9 gpus, and i think this is a bug

nvidia-smi

Sat Mar 30 02:56:08 2019 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 390.116 Driver Version: 390.116 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 P106-100 Off | 00000000:01:00.0 Off | N/A | | 29% 34C P0 28W / 120W | 0MiB / 6080MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 P106-100 Off | 00000000:03:00.0 Off | N/A | | 29% 36C P0 28W / 120W | 0MiB / 6080MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 P106-100 Off | 00000000:04:00.0 Off | N/A | | 29% 36C P0 28W / 120W | 0MiB / 6080MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 P106-100 Off | 00000000:05:00.0 Off | N/A | | 29% 35C P0 29W / 120W | 0MiB / 6080MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 4 P106-100 Off | 00000000:06:00.0 Off | N/A | | 29% 34C P0 28W / 120W | 0MiB / 6080MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 5 P106-100 Off | 00000000:07:00.0 Off | N/A | | 29% 34C P0 28W / 120W | 0MiB / 6080MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 6 P106-100 Off | 00000000:08:00.0 Off | N/A | | 29% 34C P0 28W / 120W | 0MiB / 6080MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 7 P106-100 Off | 00000000:09:00.0 Off | N/A | | 29% 33C P0 28W / 120W | 0MiB / 6080MiB | 0% Default | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

dosoos commented 5 years ago

@claudioandre-br i use your suggestion command still have this problem, why? my machine have problem?

GWS=20480 john --format=oldoffice-opencl --fork=8 --mask=?d?d?d?d?d?d?d?d xxoo.xls.hash `

GWS=20480 john --format=oldoffice-opencl --fork=8 --mask=?d?d?d?d?d?d?d?d xxoo.xls.hash

Using default input encoding: UTF-8 Loaded 1 password hash (oldoffice-opencl, MS Office <= 2003 [MD5/SHA1 RC4 OpenCL]) Cost 1 (hash type) is 0 for all loaded hashes Node numbers 1-8 of 8 (fork) 1: Local worksize (LWS) 64, global worksize (GWS) 20480 (320 blocks) 3: Local worksize (LWS) 64, global worksize (GWS) 20480 (320 blocks) 7: Local worksize (LWS) 64, global worksize (GWS) 20480 (320 blocks) 5: Local worksize (LWS) 64, global worksize (GWS) 20480 (320 blocks) Press 'q' or Ctrl-C to abort, almost any other key for status 8: Local worksize (LWS) 64, global worksize (GWS) 20480 (320 blocks) 6: Local worksize (LWS) 64, global worksize (GWS) 20480 (320 blocks) 4: Local worksize (LWS) 64, global worksize (GWS) 20480 (320 blocks) 2: Local worksize (LWS) 64, global worksize (GWS) 20480 (320 blocks) 3: Warning: Only 2120 candidates left, minimum 20480 needed for performance. 1: Warning: Only 2120 candidates left, minimum 20480 needed for performance. 1: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 3 0g 0:00:00:00 N/A 0g/s 16025Kp/s 16025Kc/s 16025KC/s GPU:40°C ##166243..##166243 7: Warning: Only 2120 candidates left, minimum 20480 needed for performance. 5: Warning: Only 2120 candidates left, minimum 20480 needed for performance.

7: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 5: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 6: Warning: Only 2120 candidates left, minimum 20480 needed for performance. 8: Warning: Only 2120 candidates left, minimum 20480 needed for performance. 6: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 8: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 4: Warning: Only 2120 candidates left, minimum 20480 needed for performance. 2: Warning: Only 2120 candidates left, minimum 20480 needed for performance. 4: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 2: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt `

john --format=oldoffice-opencl --fork=4 --mask=?d?d?d?d?d?d?d?d xxoo.xls.hash

`

john --format=oldoffice-opencl --fork=4 --mask=?d?d?d?d?d?d?d?d xxoo.xls.hash

Using default input encoding: UTF-8 Loaded 1 password hash (oldoffice-opencl, MS Office <= 2003 [MD5/SHA1 RC4 OpenCL]) Cost 1 (hash type) is 0 for all loaded hashes Node numbers 1-4 of 4 (fork) Warning: To fully use the 8 devices available, you must specify --fork=8 (see doc/README-OPENCL) 1: Local worksize (LWS) 64, global worksize (GWS) 40960 (640 blocks) 2: Local worksize (LWS) 64, global worksize (GWS) 40960 (640 blocks) 4: Local worksize (LWS) 64, global worksize (GWS) 81920 (1280 blocks) 3: Local worksize (LWS) 64, global worksize (GWS) 81920 (1280 blocks) Press 'q' or Ctrl-C to abort, almost any other key for status 2: Warning: Only 4240 candidates left, minimum 40960 needed for performance. 2 0g 0:00:00:04 N/A 0g/s 5030Kp/s 5030Kc/s 5030KC/s GPU:41°C ##154879..##154879 1: Warning: Only 4240 candidates left, minimum 40960 needed for performance. 1: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt root@gpu-server3:/usr/local/JohnTheRipper-bleeding-jumbo/run# 4: Warning: Only 4240 candidates left, minimum 81920 needed for performance. 4: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt 3: Warning: Only 4240 candidates left, minimum 81920 needed for performance. 3: OpenCL CL_OUT_OF_RESOURCES (-5) error in opencl_oldoffice_fmt_plug.c:522 - Failed transferring salt `

claudioandre-br commented 5 years ago

OpenCL has a bug now. Probably related to 24cbe35a034bd89bac325e528e367bbf2b2e59ad.

Anyway, I can't see it causing the 'CL_OUT_OF_RESOURCES' reported. Well, we are still investigating.

dosoos commented 5 years ago

Ok, I can provide this machine for you to test if necessary.

claudioandre-br commented 5 years ago

I can reproduce on super : ../run/john --format=oldoffice-opencl --fork=4 --mask=?d?d?d?d?d?d?d?d xxoo.xls.hash

But this works:

well:~/bleeding/src$ ../run/john | head -1
John the Ripper 1.8.0.16-jumbo-1-bleeding-6386664 2019-03-30 20:13:09 +0100 OMP [linux-gnu 64-bit x86_64 AVX2 AC]

super src]$ ../run/john | head -1
John the Ripper 1.8.0.16-jumbo-1-bleeding-6386664 2019-03-30 20:13:09 +0100 OMP [linux-gnu 64-bit x86_64 AVX AC]
claudioandre-br commented 5 years ago

Setting LWS=64 GWS=1024 also doesn't help.

On well you can stress even more and it handles it.

../run/john --format=oldoffice-opencl --fork=8 -dev=3,2,1,5,3,3,3,3 --mask=?d?d?d?d?d?d?d?d xxoo.xls.hash
claudioandre-br commented 5 years ago

Split into 2 sessions works on super.

../run/john --format=oldoffice-opencl --fork=3 --dev=1,1,4 --mask=?a?a?a?a?a?a?a?a xxoo.xls.hash --session=1
../run/john --format=oldoffice-opencl --fork=3 --dev=4,5,6 --mask=?a?a?a?a?a?a?a?a xxoo.xls.hash --session=2
claudioandre-br commented 5 years ago

There is an error in how JtR breaks the task into forked processes when there is not enough to do.

magnum, do you have something to say?

magnumripper commented 5 years ago

Unfortunately not. Our mask mode is good shit but I really regret we let someone write it who later disappeared for good. I simply can't understand 60% of Sayantan's code no matter how I try.

claudioandre-br commented 5 years ago

We need to start to rewrite some code (OpenCL including).

At least for mask, JtR should re-try to be in Google Summer of Code.