openwall / john

John the Ripper jumbo - advanced offline password cracker, which supports hundreds of hash and cipher types, and runs on many operating systems, CPUs, GPUs, and even some FPGAs
https://www.openwall.com/john/
Other
10.06k stars 2.08k forks source link

GPG symmetric encryption and gpg2john parsing problems #2616

Closed kholia closed 7 years ago

kholia commented 7 years ago

Setup

$ cat wordlist
1234

$ dd if=/dev/zero bs=1000000 count=1 > large

OK cases

$ gpg -c -o file_default.gpg --passphrase 1234 < large
$ ../run/gpg2john file_default.gpg > hash
$ ../run/john hash --wordlist=wordlist
Loaded 1 password hash (gpg, OpenPGP / GnuPG Secret Key [32/64])
1234             (?)

$ gpg -c --cipher-algo cast5 --force-mdc -o file_ok.gpg --passphrase 1234 < large
$ ../run/gpg2john file_ok.gpg > hash
$ ../run/john hash --wordlist=wordlist
Loaded 1 password hash (gpg, OpenPGP / GnuPG Secret Key [32/64])
1234             (?)

Not OK

GPG files lacking MDC seem to be run into parsing problems.

$ gpg -c --cipher-algo cast5 -o file_bad.gpg --passphrase 1234 < large

$ ../run/gpg2john file_bad.gpg > hash
[gpg2john] MDC is misssing, expect false positives!
unexpected end of file.

If the size and contents of "large" file are varied, gpg2john runs into multiple different problems. These gpg2john error message can also change for different runs of gpg on the same input file.

$ gpg -c --cipher-algo cast5 -o file_bad.gpg --passphrase 1234 < /bin/bash

$ ../run/gpg2john file_bad.gpg  > hash
[gpg2john] MDC is misssing, expect false positives!
Bad parameter: give_pdump(len=3004), len can not be bigger than sizeof(gecos)=1024.

For other files, the following error message is shown by gpg2john,

unknown compress algorithm.

Note: Cracking of symmetrically encrypted files lacking MDC is currently not practically possible due to too many false positives involved. See https://github.com/magnumripper/JohnTheRipper/issues/1892 for details.

kholia commented 7 years ago

The bug seems to be in Symmetrically_Encrypted_Data_Packet function (affected function). It seems that inappropriate code from Symmetrically_Encrypted_and_MDC_Packet function was inserted into the affected function.

diff --git a/src/gpg2john.c b/src/gpg2john.c
index efd8959..6517e80 100644
--- a/src/gpg2john.c
+++ b/src/gpg2john.c
@@ -934,11 +934,8 @@ Symmetrically_Encrypted_Data_Packet(int len, int first, int partial, char *hash)
        if (first) {
                cp = hash;
                totlen = 0;
-               // printf("\tVer %d\n", Getc());
-               Getc(); // version (we only read this from the first packet. Not read from rest of the 'partial' packets.
-       } else
-               ++len;  // we want the 'full' length for subsquent partial packets, since the logic is len-1 we simply fake it out.
-       totlen += (len-1);
+       }
+       totlen += (len);

        switch (mode) {
        case SYM_ALG_MODE_NOT_SPECIFIED:
@@ -963,7 +960,7 @@ Symmetrically_Encrypted_Data_Packet(int len, int first, int partial, char *hash)

        if (give(len, m_data, sizeof(m_data)) != len)
                return;
-       cp += print_hex(m_data, len - 1, cp);
+       cp += print_hex(m_data, len, cp);

        if (!partial) {
                // we only dump the packet out when we get the 'non-partial' packet (i.e. last one).

Here is a symmetrically encrypted GPG file which gets cracked only after applying this patch.

I am not sure if this patch solves the problem 100% but it is a start.

Tip: Use pgpdump (gpg2john is based on it) to rule out parsing problems with the original files.

2600box commented 7 years ago

I applied the patch and we have some functionality. I was able to test your file as well as two of my own. Your file and one of mine work. One of mine does not, because it large at 3GB.

gpg2john is creating huge files. with a small 197MB file, it becomes a 393MB hash. When I try this on a large file of 3GB it becomes a 9GB hash which john will not accept.

$ ls -lh data2*
-rw-r--r--  1 user  staff   393M  7 Jul 20:18 data2.hash
-rw-r--r--  1 user  staff   197M  7 Jul 20:27 data2.tgz.gpg

Also, I didn't know exactly how I am supposed to use pgpdump. Is it supposed to be used on the hash? pgpdump example.hash? because that just gives me pgpdump: can't find PGP armor boundary.

for interest here are some pgpdump info on my files.

3GB file:

$ pgpdump 3GB_file.tgz.gpg 
Old: Symmetric-Key Encrypted Session Key Packet(tag 3)(13 bytes)
    New version(4)
    Sym alg - CAST5(sym 3)
    Iterated and salted string-to-key(s2k 3):
        Hash alg - SHA1(hash 2)
        Salt - 07 3a 52 c3 0b a4 10 ea 
        Count - 65536(coded count 96)
New: Symmetrically Encrypted Data Packet(tag 9)(8192 bytes) partial start
    Encrypted data [sym alg is specified in sym-key encrypted session key]
New:    (8192 bytes) partial continue
New:    (4096 bytes) partial continue
New:    (2048 bytes) partial continue
New:    (512 bytes) partial continue
New:    (485 bytes) partial end

196MB file:

$ pgpdump data2.tgz.gpg 
Old: Symmetric-Key Encrypted Session Key Packet(tag 3)(13 bytes)
    New version(4)
    Sym alg - CAST5(sym 3)
    Iterated and salted string-to-key(s2k 3):
        Hash alg - SHA1(hash 2)
        Salt - d8 56 89 b0 ba f0 88 66 
        Count - 19922944(coded count 227)
New: Symmetrically Encrypted Data Packet(tag 9)(8192 bytes) partial start
    Encrypted data [sym alg is specified in sym-key encrypted session key]
New:    (8192 bytes) partial continue
New:    (4096 bytes) partial continue
New:    (2048 bytes) partial continue
New:    (512 bytes) partial continue
New:    (485 bytes) partial end

your example:

$ pgpdump example_1_1234.gpg
Old: Symmetric-Key Encrypted Session Key Packet(tag 3)(13 bytes)
    New version(4)
    Sym alg - CAST5(sym 3)
    Iterated and salted string-to-key(s2k 3):
        Hash alg - SHA1(hash 2)
        Salt - 90 bd 75 05 24 9c 47 98 
        Count - 65536(coded count 96)
New: Symmetrically Encrypted Data Packet(tag 9)(8192 bytes) partial start
    Encrypted data [sym alg is specified in sym-key encrypted session key]
New:    (1024 bytes) partial continue
New:    (512 bytes) partial continue
New:    (301 bytes) partial end

Working cracked file example here:

$ unzip example_1_1234.gpg.zip 
Archive:  example_1_1234.gpg.zip
 extracting: example_1_1234.gpg
$ ./gpg2john example_1_1234.gpg > example.hash
[gpg2john] MDC is misssing, expect false positives!
$ pgpdump example.hash 
pgpdump: can't find PGP armor boundary.
$ ls -lh example*
-rw-r--r--  1 user  staff    20K  7 Jul 20:21 example.hash
-rw-rw-r--  1 user  staff   9.8K  7 Jul 15:00 example_1_1234.gpg
-rw-r--r--@ 1 user  staff    10K  7 Jul 17:47 example_1_1234.gpg.zip
$ ./john example.hash 
Warning: detected hash type "gpg", but the string is also recognized as "gpg-opencl"
Use the "--format=gpg-opencl" option to force loading these as that type instead
Using default input encoding: UTF-8
Loaded 1 password hash (gpg, OpenPGP / GnuPG Secret Key [32/64])
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
1234             (?)
1g 0:00:00:00 DONE 2/3 (2017-07-07 20:24) 25.00g/s 12800p/s 12800c/s 12800C/s 123456..crawford
Use the "--show" option to display all of the cracked passwords reliably
Session completed
kholia commented 7 years ago

Quick note: Run pgpdump on original .gpg files, and not on hashes. Our gpg2john tool is a modified version of pgpdump. Running pgpdump won't extract hashes. I suggested running pgpdump to rule out basic parsing problems with our/your gpg files.

kholia commented 7 years ago

I don't think there is a reasonable way to handle / crack such large .gpg files currently.

However, I might have a new idea to try here. Since, there is no MDC, we actually don't need to generate a full hash which includes all 3 GB of the original data, especially when we only use 4 bytes of it, currently! We could generate a truncated hash (i.e. limited to 64 KB), and process such hashes with JtR.

The bigger problem is that cracking of symmetrically encrypted files lacking MDC is currently not practically possible due to too many false positives involved. See #1892 for details. You will run into this problem eventually.

Maybe https://github.com/kholia/PGPCrack-NG can work but it has never been tested under such "extreme" conditions.

Let us know if you find a software solution which works better for you!

2600box commented 7 years ago

Thanks for the feedback. I will test PGPCrack-NG with a very large file and see if it has any problems.

Would prefer the possibility of using JtR however.

2600box commented 7 years ago

I tested with PGPCrack-NG on a 3GB gpg cast5 without MDC and it worked as expected. No errors.

Is there an open issue for me to track progress of this idea to add the functionality you mentioned?

"We could generate a truncated hash (i.e. limited to 64 KB), and process such hashes with JtR"

kholia commented 7 years ago

Glad to hear that PGPCrack-NG is working :-)

You can keep an eye on https://github.com/magnumripper/JohnTheRipper/issues/1892 but there has been no progress on it in a long time.

I tried to understand the inner workings of PGPCrack-NG last weekend but didn't make much progress. For now, I recommend using PGPCrack-NG.

ghost commented 6 years ago

I ran into this problem. In case it helps with the bug hunt:

$ run/gpg2john 1gig_of_data.tar.gpg > /tmp/1gig_of_data.tar.gpg.jtrhash
File 1gig_of_data.tar.gpg
[gpg2john] MDC is missing, expect lots of false positives!

$ run/john --wordlist=/tmp/wdlst /tmp/1gig_of_data.tar.gpg.jtrhash
Segmentation fault

$ pgpdump 1gig_of_data.tar.gpg | uniq
Old: Symmetric-Key Encrypted Session Key Packet(tag 3)(13 bytes)
        New version(4)
        Sym alg - CAST5(sym 3)
        Iterated and salted string-to-key(s2k 3):
                Hash alg - SHA1(hash 2)
                Salt - cf ac 66 53 14 0e 72 ae 
                Count - 65536(coded count 96)
New: Symmetrically Encrypted Data Packet(tag 9)(8192 bytes) partial start
        Encrypted data [sym alg is specified in sym-key encrypted session key]
New:    (8192 bytes) partial continue
New:    (4096 bytes) partial continue
New:    (1024 bytes) partial continue
New:    (512 bytes) partial continue
New:    (191 bytes) partial end

The hash produced by gpg2john was 4gb (four times bigger than the source content).

So I started to follow the pgpcrack-ng suggestion. There's a problem building PGPCrack-NG on debian. I took these steps:

  1. $ aptitude install libpth-dev libbz2-dev libassuan-dev libgcrypt20-dev libgpg-error-dev
  2. $ cd ~/src/
  3. $ git clone https://github.com/kholia/PGPCrack-NG.git
  4. Noticed /usr/include/libassuan2 does not exist, so updated Makefile to reference the directory for libassuan.a and libassuan.so(/usr/lib/x86_64-linux-gnu/`).
  5. $ make
    gcc -I/usr/lib/x86_64-linux-gnu/ -Wall -Wno-pointer-sign -Wno-unused-but-set-variable -O3 *.c -lgcrypt -lgpg-error -lbz2 -lz -I. -o PGPCrack-NG
    compress.c:34:18: fatal error: zlib.h: No such file or directory
    #include <zlib.h>
                  ^
    compilation terminated.
    iobuf.c: In function ‘iobuf_ioctl’:
    iobuf.c:1569:7: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
       if (DBG_IOBUF)
       ^~
    iobuf.c:1573:2: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’
    if (!a && !intval && ptrval)
    ^~
    openfile.c: In function ‘overwrite_filep’:
    openfile.c:90:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
     if( cpr_enabled () )
     ^~
    openfile.c:92:2: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’
    return 1;
    ^~~~~~
    Makefile:2: recipe for target 'target' failed
    make: *** [target] Error 1

    I have the following zlib files, so I'm not sure why it's not finding them.

    linux-headers-4.9.0-4-amd64: /usr/src/linux-headers-4.9.0-4-amd64/include/config/jffs2/zlib.h
    linux-headers-4.9.0-4-amd64: /usr/src/linux-headers-4.9.0-4-amd64/include/config/pstore/zlib/compress.h
    linux-headers-4.9.0-4-amd64: /usr/src/linux-headers-4.9.0-4-amd64/include/config/squashfs/zlib.h
    linux-headers-4.9.0-4-amd64: /usr/src/linux-headers-4.9.0-4-amd64/include/config/ubifs/fs/zlib.h
    linux-headers-4.9.0-4-amd64: /usr/src/linux-headers-4.9.0-4-amd64/include/config/zlib/deflate.h
    linux-headers-4.9.0-4-amd64: /usr/src/linux-headers-4.9.0-4-amd64/include/config/zlib/inflate.h
kholia commented 6 years ago

I have the following zlib files, so I'm not sure why it's not finding them. ... linux-headers-4.9.0-4-amd64: /usr/src/linux-headers-4.9.0-4-amd64/include/config/zlib/deflate.h

These are kernel headers!

Just install the zlib-dev or a similar package to build the package on Debian. IIRC latest Debian breaks the existing project documentation.

[gpg2john] MDC is missing, expect lots of false positives!

This is the real problem and can't be solved 100%. It also has nothing to do with the build problems you report.

ghost commented 6 years ago

Thanks. I installed zlib1g-dev and it built. (zlib-dev wasn't available). You might want to add that to the readme.

kholia commented 6 years ago

I don't have the resources to maintain documentation for every new distribution. You may send patches if you care.

ghost commented 6 years ago

I'm a github novice but i'll try.

The next problem is the readme shows:

3. john -i -stdout | ./PGPCrack-NG <PGP file>

and john gives Unknown option: "-i". Is that meant to be for incremental mode, despite using a word list?

kholia commented 6 years ago

Read the updated documentation. Things evolve, and existing documentation breaks all the time. I don't have the resources to offer you any further support.

frank-dittrich commented 6 years ago

Some time ago, there was only one john option starting with letter i, --incremental, now jumbo has

--incremental         --input-encoding=     --internal-codepage=

That's why, -i is ambiguous. The error message could be more helpful in such a case. Use -inc or --inc or --incremental...