search

openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.32k stars 2.1k forks source link

JtR stops in middle of process when cracking Multibit Classic hashes #3368

Closed easyname889 closed 6 years ago

easyname889 commented 6 years ago

I am trying to crack two multibit hashes at the same time on ubuntu. It goes smoothly for a day or two then it comes to a halt. I have 3 GFORCE 1080 TI cards and some of the cards halt earlier than the other ones. Its always the same point it gets stuck at if i rerun the process, give or take. My password list is 6.3TB and are on an external USB drive and i am running JTR in TMUX session.

Im running JTR with the command ./john --wordlist=/media/noname1/8ter/bigwordfile.txt hash.txt --encoding=ISO-8859-1 --fork=3 --dev=0,1,2 --session=sweeny

Here is how it looks when it halts ./john --status=sweeny 1 0g 1:17:09:02 63,00% (ETA: 2018-08-15 22:25) 0g/s 520594p/s 1041Kc/s 1041KC/s 2 0g 0:16:01:02 21,00% (ETA: 2018-08-17 10:30) 0g/s 453713p/s 907426c/s 907426C/s 3 0g 0:17:37:01 31,00% (ETA: 2018-08-16 13:28) 0g/s 606825p/s 1213Kc/s 1213KC/s

8min later, should have bumped a couple of 0.1 percentages but nothing ./john --status=sweeny 1 0g 1:17:09:02 63,00% (ETA: 2018-08-15 22:33) 0g/s 520594p/s 1041Kc/s 1041KC/s 2 0g 0:16:01:02 21,00% (ETA: 2018-08-17 10:38) 0g/s 453713p/s 907426c/s 907426C/s 3 0g 0:17:37:01 31,00% (ETA: 2018-08-16 13:36) 0g/s 606825p/s 1213Kc/s 1213KC/s

~/john/run$ ./john --list=build-info Version: 1.8.0.13-jumbo-1-bleeding-3a0cf88 2018-07-17 07:02:20 +0530 Build: linux-gnu 64-bit x86_64 AVX2 AC OMP SIMD: AVX2, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1 CPU tests: AVX2 $JOHN is ./ Format interface version: 14 Max. number of reported tunable costs: 4 Rec file version: REC4 Charset file version: CHR3 CHARSET_MIN: 1 (0x01) CHARSET_MAX: 255 (0xff) CHARSET_LENGTH: 24 SALT_HASH_SIZE: 1048576 Max. Markov mode level: 400 Max. Markov mode password length: 30 gcc version: 5.4.0 GNU libc version: 2.23 (loaded: 2.23) OpenCL headers version: 2.0 Crypto library: OpenSSL OpenSSL library version: 01000207f OpenSSL 1.0.2g 1 Mar 2016 GMP library version: 6.1.0 File locking: fcntl() fseek(): fseek ftell(): ftell fopen(): fopen memmem(): System's

~/john/run$ ./john --list=opencl-devices Platform #0 name: NVIDIA CUDA, version: OpenCL 1.2 CUDA 9.2.147 Device #0 (0) name: GeForce GTX 1080 Device vendor: NVIDIA Corporation Device type: GPU (LE) Device version: OpenCL 1.2 CUDA Driver version: 396.37 [recommended] Native vector widths: char 1, short 1, int 1, long 1 Preferred vector width: char 1, short 1, int 1, long 1 Global Memory: 8 GB Global Memory Cache: 320 KB Local Memory: 48 KB (Local) Constant Buffer size: 64 KB Max memory alloc. size: 2 GB Max clock (MHz): 1759 Profiling timer res.: 1000 ns Max Work Group Size: 1024 Parallel compute cores: 20 CUDA cores: 2560 (20 x 128) Speed index: 4503040 Warp size: 32 Max. GPRs/work-group: 65536 Compute capability: 6.1 (sm_61) Kernel exec. timeout: yes NVML id: 0 PCI device topology: 06:00.0 PCI lanes: 1/16 Fan speed: 37% Temperature: 28°C Utilization: 29%

Device #1 (1) name:     GeForce GTX 1080
Device vendor:          NVIDIA Corporation
Device type:            GPU (LE)
Device version:         OpenCL 1.2 CUDA
Driver version:         396.37 [recommended]
Native vector widths:   char 1, short 1, int 1, long 1
Preferred vector width: char 1, short 1, int 1, long 1
Global Memory:          8 GB
Global Memory Cache:    320 KB
Local Memory:           48 KB (Local)
Constant Buffer size:   64 KB
Max memory alloc. size: 2 GB
Max clock (MHz):        1759
Profiling timer res.:   1000 ns
Max Work Group Size:    1024
Parallel compute cores: 20
CUDA cores:             2560  (20 x 128)
Speed index:            4503040
Warp size:              32
Max. GPRs/work-group:   65536
Compute capability:     6.1 (sm_61)
Kernel exec. timeout:   no
NVML id:                1
PCI device topology:    07:00.0
PCI lanes:              1/16
Fan speed:              37%
Temperature:            27°C
Utilization:            14%

Device #2 (2) name:     GeForce GTX 1080
Device vendor:          NVIDIA Corporation
Device type:            GPU (LE)
Device version:         OpenCL 1.2 CUDA
Driver version:         396.37 [recommended]
Native vector widths:   char 1, short 1, int 1, long 1
Preferred vector width: char 1, short 1, int 1, long 1
Global Memory:          8 GB
Global Memory Cache:    320 KB
Local Memory:           48 KB (Local)
Constant Buffer size:   64 KB
Max memory alloc. size: 2 GB
Max clock (MHz):        1759
Profiling timer res.:   1000 ns
Max Work Group Size:    1024
Parallel compute cores: 20
CUDA cores:             2560  (20 x 128)
Speed index:            4503040
Warp size:              32
Max. GPRs/work-group:   65536
Compute capability:     6.1 (sm_61)
Kernel exec. timeout:   no
NVML id:                2
PCI device topology:    0b:00.0
PCI lanes:              1/16
Fan speed:              37%
Temperature:            30°C
Utilization:            0%

1 1:10:36:01 Command line: ./john --restore=newrun 1 1:10:36:01 - ISO-8859-1 input encoding enabled 1 1:10:36:01 - Passwords will be stored UTF-8 encoded in .pot file 1 1:10:36:01 Sorting salts, for performance 1 1:10:36:01 - Hash type: multibit, MultiBit Wallet (min-len 0, max-len 125) 1 1:10:36:01 - Algorithm: MD5/scrypt AES 32/64 1 1:10:36:01 - Candidate passwords will be buffered and tried in chunks of 4 1 1:10:36:01 - Configured to use otherwise idle processor cycles only 2 0:11:50:01 Command line: ./john --restore=newrun 2 0:11:50:01 - ISO-8859-1 input encoding enabled 2 0:11:50:01 - Passwords will be stored UTF-8 encoded in .pot file 2 0:11:50:01 Sorting salts, for performance 2 0:11:50:01 - Will reject candidates longer than 125 characters 2 0:11:50:01 Proceeding with wordlist mode 2 0:11:50:01 - Wordlist file: /media/noname1/8ter/bigswe.txt 2 0:11:50:01 - memory mapping wordlist (6273935944705 bytes) 2 0:11:50:01 - No word mangling rules 2 0:13:44:04 - Will distribute words across nodes 1 1:10:36:01 - Will reject candidates longer than 125 characters 1 1:10:36:01 Proceeding with wordlist mode 1 1:10:36:01 - Wordlist file: /media/noname1/8ter/bigswe.txt 1 1:10:36:01 - memory mapping wordlist (6273935944705 bytes) 1 1:10:36:01 - No word mangling rules 1 1:16:57:29 - Will distribute words across nodes 1 1:17:09:02 Continuing an interrupted session 1 1:17:09:02 Loaded a total of 2 password hashes with 2 different salts 1 1:17:09:02 Sorting salts, for performance 1 1:17:09:02 Cost 1 (iteration count) is 3 for all loaded hashes 1 1:17:09:02 Cost 2 (kdf [1:MD5 2:scrypt hd 3:scrypt classic]) is 1 for all loaded hashes 1 1:17:09:02 - Node numbers 1-3 of 3 (fork) 1 1:17:09:02 Command line: ./john --restore=newrun 1 1:17:09:02 - ISO-8859-1 input encoding enabled 1 1:17:09:02 - Passwords will be stored UTF-8 encoded in .pot file 1 1:17:09:02 Sorting salts, for performance 1 1:17:09:02 - Hash type: multibit, MultiBit Wallet (min-len 0, max-len 125) 1 1:17:09:02 - Algorithm: MD5/scrypt AES 32/64 1 1:17:09:02 - Candidate passwords will be buffered and tried in chunks of 4 1 1:17:09:02 - Configured to use otherwise idle processor cycles only

What could be the problem?

kholia commented 6 years ago

This seems like a end-user support issue.

Use john-users (http://www.openwall.com/lists/john-users/) mailing list for support.

kholia commented 6 years ago

MultiBit support in JtR doesn't use GPUs (yet) I think.

solardiz commented 6 years ago

On Tue, Aug 14, 2018 at 01:48:38PM -0700, easyname889 wrote:

./john --wordlist=/media/noname1/8ter/bigwordfile.txt hash.txt --encoding=ISO-8859-1 --fork=3 --dev=0,1,2 --session=sweeny

Dhiru, we should probably refuse to run with the "--device" option when the format is neither OpenCL nor ZTEX (nor whatever future format types might be device-aware). Including for auto-detected formats, like in this example. Open a new issue for this?

magnumripper commented 6 years ago

we should probably refuse to run with the "--device" option when the format is neither OpenCL nor ZTEX

@solardiz, IIRC I was going to add a FMT_GPU (or whatever it should be called) format flag several years ago but you were against it and I never quite understood why. Without it, the above and many other things already implemented have to resort to things like strstr(format, "-opencl") || strstr(format, "-cuda") || strstr(format, "-ztex") which is kinda ugly (although it should work fine).

Perhaps we should add that flag now? I might even have misunderstood whatever you said at the time.

claudioandre-br commented 6 years ago

The user convinced me that he (or she) might be facing a bug. Although 8 minutes is not enough, there is evidence of an issue, at least for a needs info label.

After a revision of the format, I do not see how the format itself can lock; but, in my opinion, it is feasible for a library call to hang.

So, IMO it is a valid issue. And it can be a bug.

magnumripper commented 6 years ago

Note though that the user didn't press a key to get status but ran ./john --status. I wonder what a keypress would have emitted.

easyname889 commented 6 years ago

I'm running a new session now, will get back ASAP with results

easyname889 commented 6 years ago

I just checked the old session again with ./john --status The JtR status was at the same percentage as earlier but the Estimation time until finished had just been moved forward without any result showing at the percentages.

claudioandre-br commented 6 years ago

I would say magnum prefers that you move the JtR process to foreground and press a key to get the status.

Plus:

I mean, it is not hard to show that JtR is a zombie (if it really is).

magnumripper commented 6 years ago

I just checked the old session again

Sure but me thinks it's not running at all so it'll just show whatever state it was when it stopped (for whatever reason) and the ETA will move forward accordingly.

Claudio's suggestion to tail sweenty.log is a good one.

What happens of you try this?

./john --restore=sweeny

Then, after a minute or two, press some key to get a status line.

magnumripper commented 6 years ago

tail john.log

I corrected the above, it should obviously be tail sweeny.log in this case

magnumripper commented 6 years ago

BTW the log you supply in your OP is from a whole different session, called "newrun".

/me thinks we should move this discussion to john-users mailing list.

easyname889 commented 6 years ago

Actually, it is the same session, I just edited the first post due to privacy reasons.

I started a new session 1.5w ago ./john --wordlist=/media/noname1/8ter/bigswe.txt hash.txt --encoding=ISO-8859-1 --fork=3 --dev=0,1,2 --session=newrun Loaded 2 password hashes with 2 different salts (multibit, MultiBit Wallet [MD5/scrypt AES 32/64]) Cost 1 (iteration count) is 3 for all loaded hashes Cost 2 (kdf [1:MD5 2:scrypt hd 3:scrypt classic]) is 1 for all loaded hashes Warning: OpenMP was disabled due to --fork; a non-OpenMP build may be faster Node numbers 1-3 of 3 (fork) Note: This format may emit false positives, so it will keep trying even after finding a possible candidate. Press 'q' or Ctrl-C to abort, almost any other key for status 1 0g 0:00:00:08 0,00% (ETA: 2018-08-20 04:28) 0g/s 611112p/s 1222Kc/s 1222KC/s $lasamatta..$lasan 3 0g 0:00:00:08 0,00% (ETA: 2018-08-20 04:28) 0g/s 611109p/s 1222Kc/s 1222KC/s $larsen..$larskadans 2 0g 0:00:00:08 0,00% (ETA: 2018-08-20 04:30) 0g/s 610810p/s 1221Kc/s 1221KC/s $kronika..$kronikers 3 0g 0:00:00:21 0,01% (ETA: 2018-08-19 22:36) 0g/s 617381p/s 1234Kc/s 1234KC/s 1950nedmattar..1950nedmontera 1 0g 0:00:00:21 0,01% (ETA: 2018-08-19 22:36) 0g/s 617380p/s 1234Kc/s 1234KC/s 1950nedlater..1950nedlockade 2 0g 0:00:00:21 0,01% (ETA: 2018-08-19 22:36) 0g/s 617380p/s 1234Kc/s 1234KC/s 1950nedlatet..1950nedlockar Killed noname1@Pinkypromise:~/john/run$ tail newrun.log 3 0:00:00:00 - Passwords will be stored UTF-8 encoded in .pot file 3 0:00:00:00 Sorting salts, for performance 3 0:00:00:00 - Will reject candidates longer than 125 characters 3 0:00:00:00 Proceeding with wordlist mode 3 0:00:00:00 - Wordlist file: /media/noname1/8ter/bigswe.txt 3 0:00:00:00 - memory mapping wordlist (6273935944705 bytes) 3 0:00:00:00 - No word mangling rules 2 0:00:00:00 - Will distribute words across nodes 3 0:00:00:00 - Will distribute words across nodes 1 0:00:00:00 - Will distribute words across nodes noname1@Pinkypromise:~/john/run$

I now tried to restore the session noname1@Pinkypromise:~/john/run$ ./john --restore=newrun Loaded 2 password hashes with 2 different salts (multibit, MultiBit Wallet [MD5/scrypt AES 32/64]) Cost 1 (iteration count) is 3 for all loaded hashes Cost 2 (kdf [1:MD5 2:scrypt hd 3:scrypt classic]) is 1 for all loaded hashes Warning: OpenMP was disabled due to --fork; a non-OpenMP build may be faster Node numbers 1-3 of 3 (fork) Note: This format may emit false positives, so it will keep trying even after finding a possible candidate. Proceeding with wordlist:/media/noname1/8ter/bigswe.txt

A keypress in this state doesn't generate any response

easyname889 commented 6 years ago

When I run a smaller wordlist with big rules instead it runs for weeks without a problem

claudioandre-br commented 6 years ago 
./john --wordlist=/media/noname1/8ter/bigswe.txt hash.txt --encoding=ISO-8859-1 --fork=3 --dev=0,1,2 --session=newrun

Since you not using GPUs (there is no multibit OpenCL format), the --dev=0,1,2 is neither required nor used.

claudioandre-br commented 6 years ago

So, after the restore:

I can see JtR running. I'm not "out of memory", or swapping, etc.:

$ ps -eo pcpu,pid,pmem,args | sort -k 1 -r | head -10
%CPU   PID %MEM COMMAND
 918 13671  0.1 ../run/john --wordlist=wordlist /home/claudio/multibit.in --encoding=ISO-8859-1 --fork=3 --session=sweeny
 910 13669  0.0 ../run/john --wordlist=wordlist /home/claudio/multibit.in --encoding=ISO-8859-1 --fork=3 --session=sweeny
 906 13670  0.0 ../run/john --wordlist=wordlist /home/claudio/multibit.in --encoding=ISO-8859-1 --fork=3 --session=sweeny
 1.0 13707  0.0 ps -eo pcpu,pid,pmem,args
 0.0   993  0.0 [md0_raid1]
 0.0    99  0.0 [migration/24]
 0.0    98  0.0 [watchdog/23]
 0.0    97  0.0 [ksoftirqd/23]
 0.0    96  0.0 [stopper/23]
$ ps a | grep john
13601 pts/0    Rl+   15:01 ../run/john --wordlist=wordlist /home/claudio/multibit.in --encoding=ISO-8859-1 --fork=3 --session=sweeny
13602 pts/0    Rl+   15:00 ../run/john --wordlist=wordlist /home/claudio/multibit.in --encoding=ISO-8859-1 --fork=3 --session=sweeny
13603 pts/0    Rl+   15:06 ../run/john --wordlist=wordlist /home/claudio/multibit.in --encoding=ISO-8859-1 --fork=3 --session=sweeny
13651 pts/11   S+     0:00 grep john

Still running:

$ ps a | grep john
13601 pts/0    Rl+   20:49 ../run/john --wordlist=wordlist /home/claudio/multibit.in --encoding=ISO-8859-1 --fork=3 --session=sweeny
13602 pts/0    Rl+   20:50 ../run/john --wordlist=wordlist /home/claudio/multibit.in --encoding=ISO-8859-1 --fork=3 --session=sweeny
13603 pts/0    Rl+   21:01 ../run/john --wordlist=wordlist /home/claudio/multibit.in --encoding=ISO-8859-1 --fork=3 --session=sweeny
13654 pts/11   S+     0:00 grep john
magnumripper commented 6 years ago

3 0:00:00:00 - memory mapping wordlist (6273935944705 bytes)

So you are you using a 6 terabyte wordlist. I've never tested anything that crazy, but I would think it should be handled just fine (the OS pager should take care of it).

magnumripper commented 6 years ago

Killed

Was this you aborting the session, or did it terminate?

kholia commented 6 years ago

Is there anything in the output of dmesg command?

magnumripper commented 6 years ago

@easyname889 there's this setting in john.conf:

# Set this to N to disable use of memory-mapping in wordlist mode.
WordlistMemoryMap = Y

Try changing that to N instead. This should mean we revert to simply reading a line at a time from the file.

easyname889 commented 6 years ago

Was this you aborting the session, or did it terminate?

It wasn't me, it just got terminated somehow

easyname889 commented 6 years ago

Is there anything in the output of dmesg command?

[251367.931942] [ 2923]   108  2923   210378        0   241664      293             0 indicator-sessi
[251367.931943] [ 2945]   108  2945   100787        0   393216      385             0 indicator-appli
[251367.931945] [ 2962]   108  2962   161655        0   368640      895             0 pulseaudio
[251367.931945] [ 2966]   108  2966   157412        0   589824     1160             0 unity-settings-
[251367.931946] [ 3638]  1000  3638    11319        0   131072      214             0 systemd
[251367.931948] [ 3639]  1000  3639    36345        0   167936      525             0 (sd-pam)
[251367.931950] [30894]  1000 30894     9503        0   102400      133             0 tmux
[251367.931951] [30895]  1000 30895     7814        1   102400      488             0 bash
[251367.931952] [30917]  1000 30917 1531743036        0 3866759168     3414             0 john
[251367.931954] [ 1588]     0  1588    25457        0   225280      305             0 cupsd
[251367.931955] [ 1589]     0  1589    68703        0   311296      336             0 cups-browsed
[251367.931956] [ 1684]     0  1684    14799        0   147456      149             0 sshd
[251367.931957] Out of memory: Kill process 30917 (john) score 237 or sacrifice child
[251367.931961] Killed process 30917 (john) total-vm:6126972144kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[251368.422501] oom_reaper: reaped process 30917 (john), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
kholia commented 6 years ago

Is there a memory leak in MultiBit format in JtR? Or did you system simply run out of memory?

easyname889 commented 6 years ago

I really don't know how to find out that information, please guide me and I will check it out.

easyname889 commented 6 years ago

Why does this issue have the "invalid / PEBCAK" label? I don't say its wrong, but there is no proof of that the label should be used yet.

kholia commented 6 years ago

I really don't know how to find out that information, please guide me and I will check it out.

Is the memory usage of john process continually growing when cracking the MultiBit wallet (what are the values after 2 mins, 15 mins, 30 mins, and so on? You can monitor the memory usage using the top command.

If it's not then this this is a local limited problem with your system (it ran out of memory -> too many chrome tabs open?). We suspect this (or some other local end user problem) is the cause, hence the invalid label. Invalid indicates that it is not a problem with JtR itself.

kholia commented 6 years ago

Can you paste the complete OOM message from the dmesg command? In the partial OOM message, I see some large values associated with john process which are not looking good.

easyname889 commented 6 years ago

[251367.931745] Node 0 active_anon:368kB inactive_anon:120kB active_file:1032kB inactive_file:1100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:10052kB dirty:0kB writeback:0kB shmem:268kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [251367.931745] Node 0 DMA free:15880kB min:136kB low:168kB high:200kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [251367.931747] lowmem_reserve[]: 0 2145 7592 7592 7592 [251367.931749] Node 0 DMA32 free:40384kB min:19052kB low:23812kB high:28572kB active_anon:120kB inactive_anon:20kB active_file:868kB inactive_file:684kB unevictable:0kB writepending:84kB present:2322236kB managed:2256668kB mlocked:0kB kernel_stack:96kB pagetables:2203156kB bounce:0kB free_pcp:2100kB local_pcp:716kB free_cma:0kB [251367.931751] lowmem_reserve[]: 0 0 5447 5447 5447 [251367.931752] Node 0 Normal free:48128kB min:48388kB low:60484kB high:72580kB active_anon:80kB inactive_anon:296kB active_file:1388kB inactive_file:1044kB unevictable:0kB writepending:44kB present:5734400kB managed:5584464kB mlocked:0kB kernel_stack:4016kB pagetables:5345368kB bounce:0kB free_pcp:1948kB local_pcp:648kB free_cma:0kB [251367.931754] lowmem_reserve[]: 0 0 0 0 0 [251367.931756] Node 0 DMA: 24kB (U) 28kB (U) 316kB (U) 032kB 364kB (U) 2128kB (U) 0256kB 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15880kB [251367.931761] Node 0 DMA32: 3834kB (UME) 3098kB (UME) 66716kB (UME) 45932kB (UME) 16264kB (UME) 18128kB (UM) 0256kB 0512kB 01024kB 02048kB 04096kB = 42036kB [251367.931767] Node 0 Normal: 2624kB (UME) 3588kB (ME) 57616kB (UME) 35432kB (UME) 13864kB (UME) 21128kB (ME) 7256kB (UM) 4512kB (UM) 41024kB (UM) 22048kB (UM) 0*4096kB = 48008kB [251367.931773] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [251367.931774] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [251367.931774] 704 total pagecache pages [251367.931776] 11 pages in swap cache [251367.931776] Swap cache stats: add 2697099, delete 2697751, find 2083667/3683156 [251367.931777] Free swap = 7905788kB [251367.931777] Total swap = 8072188kB [251367.931777] 2018157 pages RAM [251367.931778] 0 pages HighMem/MovableOnly [251367.931778] 53903 pages reserved [251367.931778] 0 pages cma reserved [251367.931778] 0 pages hwpoisoned [251367.931779] Unreclaimable slab info: [251367.931779] Name Used Total [251367.931784] bio-2 15KB 15KB [251367.931785] kvm_async_pf 11KB 11KB [251367.931788] uvm_va_range_t 113KB 113KB [251367.931789] nvidia_stack_cache 2280KB 2424KB [251367.931790] i915_vma 15KB 15KB [251367.931791] drm_i915_gem_object 78KB 78KB [251367.931792] RAWv6 94KB 94KB [251367.931793] UDPv6 123KB 123KB [251367.931793] tw_sock_TCPv6 7KB 7KB [251367.931794] request_sock_TCPv6 15KB 15KB [251367.931795] TCPv6 127KB 127KB [251367.931796] cfq_io_cq 27KB 27KB [251367.931796] cfq_queue 35KB 35KB [251367.931797] mqueue_inode_cache 15KB 15KB [251367.931797] fuse_request 23KB 23KB [251367.931799] posix_timers_cache 15KB 15KB [251367.931800] RAW 47KB 47KB [251367.931801] tw_sock_TCP 15KB 15KB [251367.931801] request_sock_TCP 23KB 23KB [251367.931802] TCP 160KB 160KB [251367.931802] hugetlbfs_inode_cache 31KB 31KB [251367.931803] eventpoll_pwq 31KB 31KB [251367.931803] eventpoll_epi 56KB 56KB [251367.931804] request_queue 62KB 62KB [251367.931805] dmaengine-unmap-256 30KB 30KB [251367.931805] dmaengine-unmap-128 31KB 31KB [251367.931806] file_lock_cache 15KB 15KB [251367.931807] net_namespace 89KB 89KB [251367.931808] shmem_inode_cache 1151KB 1151KB [251367.931808] taskstats 30KB 30KB [251367.931809] sigqueue 15KB 15KB [251367.931809] kernfs_node_cache 4410KB 4410KB [251367.931810] mnt_cache 165KB 165KB [251367.931826] filp 781KB 1032KB [251367.931827] lsm_file_cache 171KB 183KB [251367.931828] nsproxy 39KB 39KB [251367.931828] vm_area_struct 2045KB 2045KB [251367.931829] mm_struct 464KB 464KB [251367.931829] files_cache 158KB 158KB [251367.931830] signal_cache 368KB 368KB [251367.931831] sighand_cache 585KB 618KB [251367.931835] task_struct 1569KB 1976KB [251367.931839] cred_jar 220KB 381KB [251367.931841] anon_vma 426KB 446KB [251367.931846] pid 452KB 520KB [251367.931847] Acpi-Operand 594KB 594KB [251367.931847] Acpi-ParseExt 15KB 15KB [251367.931848] Acpi-State 31KB 31KB [251367.931848] Acpi-Namespace 346KB 346KB [251367.931849] numa_policy 15KB 15KB [251367.931850] trace_event_file 177KB 177KB [251367.931851] ftrace_event_field 348KB 366KB [251367.931852] task_group 63KB 63KB [251367.931854] kmalloc-8192 2632KB 2752KB [251367.931855] kmalloc-4096 1776KB 1888KB [251367.931856] kmalloc-2048 3114KB 3232KB [251367.931858] kmalloc-1024 3270KB 3328KB [251367.931860] kmalloc-512 1793KB 1880KB [251367.931865] kmalloc-256 359KB 412KB [251367.931866] kmalloc-192 809KB 815KB [251367.931867] kmalloc-128 278KB 288KB [251367.931868] kmalloc-96 637KB 637KB [251367.931874] kmalloc-64 769KB 884KB [251367.931876] kmalloc-32 411KB 456KB [251367.931877] kmalloc-16 124KB 124KB [251367.931877] kmalloc-8 56KB 56KB [251367.931878] kmem_cache_node 20KB 20KB [251367.931878] kmem_cache 70KB 70KB [251367.931878] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [251367.931884] [ 258] 0 258 8849 0 98304 111 0 systemd-journal [251367.931886] [ 283] 0 283 11369 0 114688 348 -1000 systemd-udevd [251367.931888] [ 733] 100 733 25596 0 102400 77 0 systemd-timesyn [251367.931889] [ 763] 0 763 1099 0 53248 43 0 acpid [251367.931891] [ 766] 106 766 11056 1 126976 396 -900 dbus-daemon [251367.931892] [ 800] 0 800 43758 0 172032 199 0 thermald [251367.931893] [ 804] 0 804 116063 0 327680 791 0 NetworkManager [251367.931894] [ 818] 0 818 74995 0 212992 295 0 accounts-daemon [251367.931895] [ 819] 104 819 64098 0 131072 315 0 rsyslogd [251367.931896] [ 820] 0 820 77679 0 188416 2754 -900 snapd [251367.931897] [ 821] 0 821 9401 0 110592 75 0 cron [251367.931898] [ 832] 0 832 7175 0 98304 130 0 systemd-logind [251367.931900] [ 889] 0 889 4892 0 73728 79 0 irqbalance [251367.931901] [ 942] 0 942 73040 0 204800 272 0 lightdm [251367.931902] [ 944] 0 944 76525 0 233472 763 0 polkitd [251367.931903] [ 953] 0 953 16377 0 163840 186 -1000 sshd [251367.931904] [ 1063] 0 1063 4031 0 77824 214 0 dhclient [251367.931905] [ 1075] 65534 1075 15365 0 155648 107 0 dnsmasq [251367.931906] [ 1187] 111 1187 11227 0 126976 117 0 avahi-daemon [251367.931907] [ 1188] 111 1188 11195 0 122880 83 0 avahi-daemon [251367.931908] [ 1560] 0 1560 88959 0 253952 412 0 upowerd [251367.931909] [ 1646] 118 1646 45886 0 122880 96 0 rtkit-daemon [251367.931911] [ 1658] 113 1658 80531 1 253952 718 0 colord [251367.931912] [ 1772] 0 1772 95599 0 233472 616 0 udisksd [251367.931913] [ 1847] 109 1847 94604 0 364544 473 0 whoopsie [251367.931914] [ 1852] 0 1852 6133 0 77824 38 0 agetty [251367.931915] [ 2811] 0 2811 62854 2250 364544 2851 0 Xorg [251367.931917] [ 2823] 122 2823 4285 0 86016 45 0 nvidia-persiste [251367.931919] [ 2858] 0 2858 56545 0 225280 204 0 lightdm [251367.931920] [ 2861] 108 2861 11319 0 131072 213 0 systemd [251367.931921] [ 2862] 108 2862 36345 0 167936 525 0 (sd-pam) [251367.931922] [ 2869] 108 2869 1126 0 57344 23 0 lightdm-greeter [251367.931923] [ 2874] 108 2874 10749 0 122880 165 0 dbus-daemon [251367.931925] [ 2875] 108 2875 234177 0 729088 2881 0 unity-greeter [251367.931926] [ 2877] 108 2877 88432 0 200704 201 0 at-spi-bus-laun [251367.931927] [ 2881] 108 2881 70753 1 176128 179 0 gvfsd [251367.931928] [ 2886] 108 2886 88607 0 188416 230 0 gvfsd-fuse [251367.931929] [ 2893] 108 2893 10691 0 135168 97 0 dbus-daemon [251367.931930] [ 2898] 108 2898 51740 0 172032 171 0 at-spi2-registr [251367.931932] [ 2905] 108 2905 44633 0 122880 131 0 dconf-service [251367.931933] [ 2910] 0 2910 20677 1 200704 144 0 lightdm [251367.931934] [ 2913] 108 2913 13638 1 135168 170 0 upstart [251367.931935] [ 2915] 108 2915 167581 1 622592 1418 0 nm-applet [251367.931936] [ 2917] 108 2917 94668 0 237568 316 0 indicator-messa [251367.931937] [ 2918] 108 2918 89409 0 200704 213 0 indicator-bluet [251367.931938] [ 2919] 108 2919 92024 0 212992 437 0 indicator-power [251367.931939] [ 2920] 108 2920 136732 0 344064 499 0 indicator-datet [251367.931940] [ 2921] 108 2921 143615 0 548864 1523 0 indicator-keybo [251367.931941] [ 2922] 108 2922 171104 0 319488 494 0 indicator-sound [251367.931942] [ 2923] 108 2923 210378 0 241664 293 0 indicator-sessi [251367.931943] [ 2945] 108 2945 100787 0 393216 385 0 indicator-appli [251367.931945] [ 2962] 108 2962 161655 0 368640 895 0 pulseaudio [251367.931945] [ 2966] 108 2966 157412 0 589824 1160 0 unity-settings- [251367.931946] [ 3638] 1000 3638 11319 0 131072 214 0 systemd [251367.931948] [ 3639] 1000 3639 36345 0 167936 525 0 (sd-pam) [251367.931950] [30894] 1000 30894 9503 0 102400 133 0 tmux [251367.931951] [30895] 1000 30895 7814 1 102400 488 0 bash [251367.931952] [30917] 1000 30917 1531743036 0 3866759168 3414 0 john [251367.931954] [ 1588] 0 1588 25457 0 225280 305 0 cupsd [251367.931955] [ 1589] 0 1589 68703 0 311296 336 0 cups-browsed [251367.931956] [ 1684] 0 1684 14799 0 147456 149 0 sshd [251367.931957] Out of memory: Kill process 30917 (john) score 237 or sacrifice child [251367.931961] Killed process 30917 (john) total-vm:6126972144kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [251368.422501] oom_reaper: reaped process 30917 (john), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

solardiz commented 6 years ago

This looks like a case of exhausting RAM through page tables in the kernel. Without huge pages, mapping a 6 TB file (and actually accessing those pages) should need 12 GB RAM (maybe more with multiple forked processes), whereas the system appears to have only 8 GB (plus swap, but it won't help in this case). I wonder why transparent huge pages presumably don't (fully) kick in.

What the user may do as a workaround is either avoid using wordlists this large (obvious) or disable use of mmap for wordlists (WordlistMemoryMap = N in john.conf).

What we may do is explicitly use huge pages, with a fallback if that fails, like I do in yescrypt-platform.c. Edit: Oh, I guess that won't work for memory-mapped files, and is why transparent huge pages also don't work. Then there's little we can do, short of being aware of this "problem" and making our users aware. Well, maybe we can introduce a WordlistMemoryMapMaxSize or whatever, and set the default to, say, 1 TB, so that for larger wordlists the memory-mapping would be automatically disabled?

magnumripper commented 6 years ago

@easyname889 a 6 GB wordlist should work on most systems but is still insane IMHO and strongly indicates poor understanding of your tools and/or poor quality of the wordlist. And your wordlist is a THOUSAND TIMES that. I do regard it a "PEBCAK" but it's not really meant as an offense if you thought so. Having said that, if you do change that setting in john.conf we've mentioned and things STILL go wrong - then we do have a bug.

maybe we can introduce a WordlistMemoryMapMaxSize or whatever, and set the default to, say, 1 TB, so that for larger wordlists the memory-mapping would be automatically disabled?

That is a good idea, I'll look into that.

kholia commented 6 years ago

A 6 GB wordlist should work on most systems...

According to @solardiz, the wordlist size seems to be 6 TB?

solardiz commented 6 years ago

Maybe we need to drop "PEBCAK" from the label name, leaving just "invalid". There's no need to use words that could appear offensive.

Regarding a WordlistMemoryMapMaxSize numeric setting, I think it could replace the existing WordlistMemoryMap Boolean setting, where 0 size would correspond to having this functionality disabled (same as WordlistMemoryMap = N). Maybe the numeric value should be in MB.

magnumripper commented 6 years ago

A 6 GB wordlist should work on most systems...

According to @solardiz, the wordlist size seems to be 6 TB?

Sure, read my text again.

kholia commented 6 years ago

Maybe we need to drop "PEBCAK" from the label name, leaving just "invalid". There's no need to use words that could appear offensive.

Ack. This has been fixed now.