Forked threads 2-N all process the same subset of the wordlist.

[farmerchris]

I have found that when I fork more than 2 processes, the 2nd through Nth all operate on the exact same set of words from the word list. I ran the command below on a hash file with 1 record remaining to crack. The wordlist is sorted alphabetically. The first thread proceeded through the alphabet from the d's to the e's, etc. The remaining 3 threads all worked on the b's.

Looking at the output .rec files, it seems there is no difference between the contents of rec 2 through N. When restored, they are all reading the same instructions and therefore processing the same passwords.

Attach details about your OS and about john, including:

• The output of ./john --list=build-info.
• The command line you are using.

> nice John myhash.hash --fork=4 --session=sess -w=rockyouaz.sorted --max-length=4
Warning: detected hash type "bcrypt", but the string is also recognized as "bcrypt-opencl"
Use the "--format=bcrypt-opencl" option to force loading these as that type instead
Using default input encoding: UTF-8
Loaded 4 password hashes with 4 different salts (bcrypt [Blowfish 32/64 X2])
Remaining 1 password hash
Cost 1 (iteration count) is 4096 for all loaded hashes
Node numbers 1-4 of 4 (fork)
Press 'q' or Ctrl-C to abort, almost any other key for status
...
1 0g 0:00:01:32 22.55% (ETA: 16:10:51) 0g/s 6.268p/s 6.268c/s 6.268C/s dies..diez
2 0g 0:00:01:32 8.63% (ETA: 16:21:49) 0g/s 2.266p/s 2.266c/s 2.266C/s bazy..bb
3 0g 0:00:01:32 8.68% (ETA: 16:21:43) 0g/s 2.277p/s 2.277c/s 2.277C/s bbg..bbk
4 0g 0:00:01:32 8.58% (ETA: 16:21:56) 0g/s 2.254p/s 2.254c/s 2.254C/s baum..baya
1 0g 0:00:01:43 24.57% (ETA: 16:11:03) 0g/s 6.274p/s 6.274c/s 6.274C/s e..eami
2 0g 0:00:01:35 8.71% (ETA: 16:22:14) 0g/s 2.278p/s 2.278c/s 2.278C/s bbma..bbum
3 0g 0:00:01:35 8.75% (ETA: 16:22:09) 0g/s 2.287p/s 2.287c/s 2.287C/s bcl..bdbd
4 0g 0:00:01:35 8.70% (ETA: 16:22:15) 0g/s 2.266p/s 2.266c/s 2.266C/s bbim..bbq

./john --list=build-info
Version: 1.9.0-jumbo-1+bleeding-ffd18e6b5d 2024-09-29 04:20:54 +0200
Build: darwin23.5.0 64-bit arm ASIMD AC OPENCL
SIMD: ASIMD, interleaving: MD4:2 MD5:2 SHA1:1 SHA256:1 SHA512:1
$JOHN is ./
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 2147483648
SINGLE_BUF_MAX: 4294967295
Effective limit: Number of salts vs. SingleMaxBufferSize
Max. Markov mode level: 400
Max. Markov mode password length: 30
clang version: 15.0.0 (clang-1500.3.9.4) (gcc 4.2.1 compatibility)
OpenCL headers version: 1.2
Crypto library: None
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
times(2) sysconf(_SC_CLK_TCK) is 100
Using times(2) for timers, resolution 10 ms
HR timer: mach_absolute_time(), resolution 42 ns
Total physical host memory: 8 GiB
Available physical host memory: 1252 MiB
Terminal locale string: en_US.UTF-8
Parsed terminal locale: UTF-8

[farmerchris]

I have seen cases where different processes are printed as processing overlapping word ranges:

nice john my.hash --fork=6 --session=sess -w=rockyouaz.sorted --max-length=4
5 0g 0:00:01:18 8.43% (ETA: 16:31:18) 0g/s 1.767p/s 1.767c/s 1.767C/s base..bash
4 0g 0:00:01:32 8.49% (ETA: 16:33:56) 0g/s 1.726p/s 1.726c/s 1.726C/s bart..bata

[farmerchris]

1 0g 0:00:01:32 22.55% (ETA: 16:10:51) 0g/s 6.268p/s 6.268c/s 6.268C/s dies..diez 2 0g 0:00:01:32 8.63% (ETA: 16:21:49) 0g/s 2.266p/s 2.266c/s 2.266C/s bazy..bb 3 0g 0:00:01:32 8.68% (ETA: 16:21:43) 0g/s 2.277p/s 2.277c/s 2.277C/s bbg..bbk 4 0g 0:00:01:32 8.58% (ETA: 16:21:56) 0g/s 2.254p/s 2.254c/s 2.254C/s baum..baya

Looking more at this I see it may be that the 1st thread is just moving faster through the list. It has more p/s and c/s than the other threads. Any reason this would be expected?

[farmerchris]

I think this is not a real issue.

[solardiz]

We assume the wordlist might be sorted for decreasing word frequency or estimated probability, which is why we don't just split it in pieces like first quarter and on, but instead have all processes start at the beginning and skip other processes' words. So it's fine that the ranges in an alphabetically-sorted wordlist appear overlapping - this does not mean that the processes are trying the same candidate passwords.

Looking more at this I see it may be that the 1st thread is just moving faster through the list. It has more p/s and c/s than the other threads. Any reason this would be expected?

Such substantial difference in performance is generally unexpected. My guess is that somehow only one process is running on a performance core and the rest on efficiency cores in your Apple Silicon CPU. I'm not familiar with macOS process scheduler and don't know whether it would commonly maintain CPU affinity like that instead of juggling processes between core types. Maybe for power efficiency reasons it's by default unwilling to give one parent task more than one performance core, or something. @magnumripper might have a better idea.

[farmerchris]

Thanks for clarifying. after studying some more I’m satisfied that the behavior I was seeing is caused by reduced cpu assigned to forked processes vs the parent process. I don’t know why that would happen but it doesn’t seem to be a john bug so I’ll retract the issue. - ChrisOn Sep 29, 2024, at 6:38 PM, Solar Designer @.***> wrote: We assume the wordlist might be sorted for decreasing word frequency or estimated probability, which is why we don't just split it in pieces like first quarter and on, but instead have all processes start at the beginning and skip other processes' words. So it's fine that the ranges in an alphabetically-sorted wordlist appear overlapping - this does not mean that the processes are trying the same candidate passwords.

Looking more at this I see it may be that the 1st thread is just moving faster through the list. It has more p/s and c/s than the other threads. Any reason this would be expected?

Such substantial difference in performance is generally unexpected. My guess is that somehow only one process is running on a performance core and the rest on efficiency cores in your Apple Silicon CPU. I'm not familiar with macOS process scheduler and don't know whether it would commonly maintain CPU affinity like that instead of juggling processes between core types. Maybe for power efficiency reasons it's by default unwilling to give one parent task more than one performance core, or something. @magnumripper might have a better idea.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you modified the open/close state.Message ID: @.***>