problems generating hashes from archives

[BotPass]

The tools for generating hashes out of archives may create invalid files, depending on the names and paths of the files. The tools use colons to separate different parts. But colons may be part of the file name. This is especially true for absolute names under Windows. The tools write the file name (for rar5 the given name --> probably absolute file name) at the front of the hash line and the given name at the end. With colon as part of these, the separation is messed up. I come up with this because I use absolute names under Windows and my rar5 extraction were all broken.

How to reproduce? For reproduction I used Linux with a build of a current github version (ea33667 from Sep 23rd).

generating archive files I used the password "test":

me@test:/jtr_ea33667/test$ dd if=/dev/urandom of=testfile.dd bs=4K count=2
2+0 records in
2+0 records out
8192 bytes (8.2 kB, 8.0 KiB) copied, 0.000401248 s, 20.4 MB/s

me@test:/jtr_ea33667/test$ zip -e \(\:\).zip testfile.dd 
Enter password: 
Verify password: 
  adding: testfile.dd (stored 0%)

me@test:/jtr_ea33667/test$ 7z a -p \(\:\).7z testfile.dd 

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 ([...])

Scanning the drive:
1 file, 8192 bytes (8 KiB)

Creating archive: (:).7z

Items to compress: 1

Enter password (will not be echoed):
Verify password (will not be echoed) :

Files read from disk: 1
Archive size: 8354 bytes (9 KiB)
Everything is Ok

me@test:/jtr_ea33667/test$ rar a -p \(\:\).rar testfile.dd 

Enter password (will not be echoed): 

Reenter password: 

RAR 5.50   Copyright (c) 1993-2017 Alexander Roshal   11 Aug 2017
[...]

Creating archive (:).rar

Adding    testfile.dd                                                 OK 
Done
me@test:/jtr_ea33667/test$

extracting hashes

me@test:/jtr_ea33667/test$ ../run/zip2john *.zip > zip.hash
ver 1.0 efh 5455 efh 7875 (:).zip/testfile.dd PKZIP Encr: 2b chk, TS_chk, cmplen=8204, decmplen=8192, crc=82C9D06C
me@test:/jtr_ea33667/test$ ../run/rar2john *.rar > rar.hash
me@test:/jtr_ea33667/test$ ../run/7z2john.pl *.7z > 7z.hash

use john

test@test:/jtr_ea33667/test$ ../run/john  7z.hash --mask=test
Using default input encoding: UTF-8
No password hashes loaded (see FAQ)

test@test:/jtr_ea33667/test$ ../run/john  rar.hash --mask=test
Using default input encoding: UTF-8
No password hashes loaded (see FAQ)

test@test:/jtr_ea33667/test$ ../run/john  zip.hash --mask=test
Using default input encoding: UTF-8
No password hashes loaded (see FAQ)

As you can see, john is not able to find a hash in the files created by the extraction tools of john the ripper.

possible solution I thought of at least four different "solutions", but I forget at least one. Thus here are 3 possible options I thought of:

a) escaping problematic chars in the generated hash line and decode escaping while reading these lines

b) simply remove problematic chars; but if the names are used later they cannot be reconstructed correctly

c) prohibit the use of problematic chars like colon; if an extraction tool finds a problem, it should abort and deny creation of hash; thus the user had to rename the files and Windows user are not permitted to use absolute file names

additional

I have additional short "problems" partly not related to this. But I do not think they are worth another issue report and some are only notes:

'1) in rar2john.c lines 210-211:

    strnzcpy(path, archive_name, sizeof(path));
    base_aname = basename(path);

As far as I can tell these are the only lines in this function with the local variable path ("char path[PATH_BUFFER_SIZE]") with "#define PATH_BUF_SIZE 256". The variable "archive_name" seems to be the original file name given to rar2john. Depending on OS this could be longer than 256 bytes (e.g. 4096 on some Linux; 32k for extended-length path on Windows). Thus "base_aname" is not necessarily the basename. I do not know why it is not simply "basename(archive_name)".

'2) In one case zip2john's result was notified by john with utf-8 encoding error, because an existing file name from archive had strange encoding (not tested with the current build above).

'3) in opencl_rawsha512.h lines 40-42 "uint" is used, but not known by some mingw compiler (no official data type).

[claudioandre-br]

The bug affects *NIX, I agree. But, not Windows. On Windows the colon is not valid as a filename; it is valid as a driver only. And JtR can handle that.

• Hash generated on Windows (with a driver letter) winhash.txt

  ---

  Crackable on Windows and Linux

  
  C:\>Temp\JohnTheRipper\run\john c:\Temp\winhash.txt
  Warning: detected hash type "ZIP", but the string is also recognized as "ZIP-opencl"
  Use the "--format=ZIP-opencl" option to force loading these as that type instead

Using default input encoding: UTF-8 Loaded 1 password hash (ZIP, WinZip [PBKDF2-SHA1 128/128 SSE2 4x2]) Will run 2 OpenMP threads Proceeding with single, rules:Single Press 'q' or Ctrl-C to abort, almost any other key for status Warning: Only 1 candidate buffered for the current salt, minimum 16 needed for performance. test (teste.zip/HPSF_Rep.txt) 1g 0:00:00:00 DONE 1/3 (2019-10-13 13:20) 1.424g/s 1026p/s 1026c/s 1026C/s test..reph Use the "--show" option to display all of the cracked passwords reliably Session completed

$ john winhash.txt Using default input encoding: UTF-8 Loaded 1 password hash (ZIP, WinZip [PBKDF2-SHA1 128/128 SSE2 4x]) Will run 2 OpenMP threads Proceeding with single, rules:Single Press 'q' or Ctrl-C to abort, almost any other key for status test (teste.zip/HPSF_Rep.txt) 1g 0:00:00:00 DONE 1/3 (2019-10-13 13:43) 1.754g/s 1408p/s 1408c/s 1408C/s Reptest..zipt Use the "--show" option to display all of the cracked passwords reliably Session completed

> '3)
> in opencl_rawsha512.h lines 40-42
> "uint" is used, but not known by some mingw compiler (no official data type).

There is a `type_def`. Can you see any error message in your compiler?

[BotPass]

Well I first experienced this problem under Windows. The Linux example is just some thing I created for testing. conditions for Windows: rar5 and absolute name

C:\jtr_ea33667\run>type \\?\c:\e\example.rar
Rar!
C:\jtr_ea33667\run>rar2john.exe \\?\c:\e\example.rar

C:\jtr_ea33667\run>! \\example.rar: No such file or directory

C:\jtr_ea33667\run>rar2john.exe c:\e\example.rar

C:\jtr_ea33667\run>c:\e\example.rar:$rar5$16$[...]

As far as I can tell the given file name is always written into the hash line. In rar5 it is the first element and thus before the hash itself. In rar3/4 7z and pkzip it is after the hash

[short form]:[hash]:[...]:c:\e\example.[...]

I do not know if this part is ever parsed. Thus I do not know if this is also a problem for Windows user. But yes, up to now I only experienced the "no hash loaded" problem under Windows only for rar5.

---

opencl_rawsha512.h)

My compiler failed building cause of unknown uint. That's why I bring it up. I greped for type_def but could not find anything. Where should it be? What did I missed?

[claudioandre-br]

Where should it be? What did I missed?

opencl_rawsha512.h -> opencl_sha512.h -> opencl_sha2_common.h -> opencl_misc.h

BTW: they are inside run/opencl.

[magnumripper]

It's not type_def but typedef

$ git grep -P '^typedef uint ' ../run/opencl/
../run/opencl/opencl_aes_plain.h:typedef uint u32;
../run/opencl/opencl_misc.h:typedef uint uint32_t;
../run/opencl/opencl_twofish.h:typedef uint Twofish_UInt32;
../run/opencl/opencl_unicode.h:typedef uint   UTF32;

[solardiz]

I think we fixed this same issue as #4200. I've just confirmed that the corresponding fix commit includes changes to rar2john.