search

openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.2k stars 2.09k forks source link

zip2john: Misbehavior with very long filenames #5174

Closed bf closed 1 year ago

bf commented 2 years ago

I'm using zip2john on a windows-created zip archive which includes a directory and within that directory there are two encrypted files. It seems the directory name is bugged/malformed, as it ends with a backslash symbol (7z l and unzip -l both show this).

Now the problem is, that zip2john crashes with the following error message when the zip archive name is too long:

$ zip2john -o "111 Ab 1111 - Anlage zu Schreiben von Abbbbbb cccccccc vom 99.99.9999\\Excel-Tabelle 2005.xlsx" c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7 
Using file 111 Ab 1111 - Anlage zu Schreiben von Abbbbbb cccccccc vom 99.99.9999\Excel-Tabelle 2005.xlsx as only file to check
c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/111 Ab 1111 � Anlage zu Schreiben von Abbbbbb cccccccc vom 99.99.9999/ is not encrypted!
malloc(): corrupted top size
Aborted (core dumped)

Now it gets funny. When I shorten filename of zip file via mv c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7 c5bb or even I create a short symlink to the zip file via ln -s c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7 foo then the zip2john runs as expected, outputting the hashes.

Please find version info attached. I'm on arch linux.

I can share the core dump via email if it helps you.

Version: 1.9.0-jumbo-1
Build: linux-gnu 64-bit x86_64 AVX AC MPI + OMP
SIMD: AVX, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
System-wide exec: /usr/lib/john
System-wide home: /usr/share/john
Private home: ~/.john
CPU tests: AVX
CPU fallback binary: john-non-avx
$JOHN is /usr/share/john/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 2147483648
SINGLE_BUF_MAX: 4294967295
Effective limit: Number of salts vs. SingleMaxBufferSize
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 12.1.0
GNU libc version: 2.35 (loaded: 2.35)
OpenCL headers version: 1.2
Crypto library: OpenSSL
OpenSSL library version: 0101010ff  (loaded: 01010111f)
OpenSSL 1.1.1o  3 May 2022  (loaded: OpenSSL 1.1.1q  5 Jul 2022)
GMP library version: 6.2.1
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
solardiz commented 2 years ago

Thank you for reporting this. There have been so many commits changing zip2john.c since the 1.9.0-jumbo-1 release that I can't easily tell whether we still have this issue or maybe not. My quick attempt at reproducing it failed so far. Next is to look at the code.

Given that those commits include important fixes and we'd recommend that you use the latest code anyway, would you please try building the latest from this repository and using it on your archive (both zip2john and john)? Thanks!

solardiz commented 2 years ago

Well, old code (as in 1.9.0-jumbo-1) limited filenames to 250 chars and allocated 400 chars beyond compressed data size in hex. So could the rest of what's put into the "hash" exceed 150? Anyway, this code has since been rewritten.

bf commented 2 years ago

Thanks for your immediate answer. I'm currently building latest git and will try. (So far it seems it works with new version in the dev environment).

bf commented 2 years ago

With git-compiled version it is not crashing any more, but also not returning any output.

It only says Using file fooo\bar.xlsx as only file to check and then stops with return code 0.

solardiz commented 2 years ago

Maybe @srett and/or @magnumripper would look into this further, if you share a file reproducing this issue with one or both of them.

magnumripper commented 2 years ago

Yeah, please send it to magnumripper at hushmail.com and I'll have a look.

bf commented 2 years ago

I cannot share original file and failed to build another zip file with the same error.

But DEFINITELY the problem must be related to the use of backslashes as path separators within the zip file. On linux I simply cannot create zip files which use backslash (which is also against zip standard afaik).

srett commented 2 years ago

But DEFINITELY the problem must be related to the use of backslashes as path separators within the zip file. On linux I simply cannot create zip files which use backslash (which is also against zip standard afaik).

From a quick glance at the code, there doesn't seem to be any place in particular that cares about the name of the files in the archive. It's treated as opaque values and just used for printing, so it should work with whatever crazy/malformed file name you can come up with. I mean, to test it, take the zip archive created on Linux, and replace / with \ in all occurrences of the file name.

Maybe someone can cobble together a script/tool that just dumps the raw local+central headers from the zip file, but not the actual file payloads, or wipes out those parts. That should result in a shareable zip file that can never be extracted, even with the correct password.

Btw. I left a bunch of debug output in the code that might, with a little bit of luck, help find the issue. For that the code has to be compiled with -DDEBUG, I just don't remember whether that will get turned on with some switch to ./configure or whether I just ran the according gcc call manually with that thrown in.

magnumripper commented 2 years ago

Btw. I left a bunch of debug output in the code that might, with a little bit of luck, help find the issue. For that the code has to be compiled with -DDEBUG, I just don't remember whether that will get turned on with some switch to ./configure or whether I just ran the according gcc call manually with that thrown in.

make debug will do: It adds -DDEBUG and also decreases optimization level to -O1 (or -Og, if supported). So given a non-debug build already made, you can touch zip2john.c ; make -s debug to get just zip2john with debug options.

magnumripper commented 2 years ago

But DEFINITELY the problem must be related to the use of backslashes as path separators within the zip file. On linux I simply cannot create zip files which use backslash (which is also against zip standard afaik).

You said earlier that just renaming the archive (or symlinking) would make it work and produce a hash - but that was before upgrading to latest code. That indicates the contents of the archive isn't (or wasn't) the problem. This would be a hint, if it wasn't for the amount of changes in the code...

bf commented 2 years ago

Here is a valid zip with the "filename too long" error. I think the malloc() was triggered by some UTF16 in internal filenames (see original comment).

foo.zip

password is "test"

claudioandre-br commented 2 years ago

It works here (Ubuntu 22 LTS). Do you have a VM or docker (so you can) run some tests using your official file?

$ john-the-ripper.zip2john foo.zip > hash.in
ver 2.0 foo.zip/c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/ is not encrypted, or stored with non-handled compression type
ver 2.0 foo.zip/c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/ is not encrypted, or stored with non-handled compression type
ver 2.0 foo.zip/c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/ is not encrypted, or stored with non-handled compression type
ver 2.0 foo.zip/c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/ is not encrypted, or stored with non-handled compression type
$ john hash.in 
Warning: detected hash type "ZIP", but the string is also recognized as "ZIP-opencl"
Use the "--format=ZIP-opencl" option to force loading these as that type instead
Using default input encoding: UTF-8
Loaded 1 password hash (ZIP, WinZip [PBKDF2-SHA1 256/256 AVX2 8x])
Cost 1 (HMAC size) is 695302 for all loaded hashes
Will run 8 OpenMP threads
Proceeding with single, rules:Single
Press 'q' or Ctrl-C to abort, 'h' for help, almost any other key for status
Almost done: Processing the remaining buffered candidate passwords, if any.
0g 0:00:00:02 DONE 1/3 (2022-10-10 09:07) 0g/s 25612p/s 25612c/s 25612C/s Bzip1900..Png1900
Proceeding with wordlist:/snap/john-the-ripper/current/run/password.lst
Enabling duplicate candidate password suppressor
test             (foo.zip/c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7.png)     
1g 0:00:00:03 DONE 2/3 (2022-10-10 09:07) 0.3311g/s 22386p/s 22386c/s 22386C/s 123456..abundance
Use the "--show" option to display all of the cracked passwords reliably
Session completed. 
$ unzip -l foo.zip 
Archive:  foo.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2022-10-05 16:19   c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/
        0  2022-10-05 16:21   c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/
        0  2022-10-05 16:21   c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/
        0  2022-10-05 16:21   c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/
   703628  2019-11-17 15:10   c5b b96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7/c5bb96420536d18bf8a052771384a09a2ee7040a00a9c2d6cbc8a2e2ee0876237ae1ee65b0e7.png
---------                     -------
   703628                     5 files
claudioandre-br commented 2 years ago

If you still think john has a bug, you can have reproducible and repeatable runs:

Current folder has the file foo.zip.

docker run -it --entrypoint=bash --rm -v $(pwd):/host ghcr.io/openwall/john:rolling
# you are now "INSIDE" Docker
cd /host
ln -s /john/run/john-avx zip2john
./zip2john foo.zip

: image

noraj commented 1 year ago

In my case (not even a long name) it's broken on last 1.9.0-jumbo-1 release but works on latest rolling docker image (sha256:9300e6e6433ce2313d96fcbd8c6957f778703ce260f9d4e087b921b0840d94ad).

magnumripper commented 1 year ago

Yeah I can't reproduce any problem with latest bleeding zip2john and the supplied sample. I even tried hex-editing it to have backslash separators and there were still no problems getting a crackable hash - although that wasn't a 100% reliable test because the edited archive was detected as b0rken by zip tools.

magnumripper commented 1 year ago

I even tried hex-editing it to have backslash separators

According to https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE_6.2.0.txt, the path separators will be / even on archives created on DOS/Windows - it's just their screen output that differs.

The path stored should not contain a drive or device letter, or a leading slash. All slashes should be forward slashes '/' as opposed to backwards slashes '\' for compatibility with Amiga and Unix file systems etc. If input came from standard input, there is no file name field.

I regard this as fixed. @bf if you have reproducible problems using latest code, please reopen.