symmetric gpg fails to decrypt with a large input and mdc on

[fNBU]

If I encrypt a large file using gpg with symmetric encryption and --force-mdc, and defaults otherwise, john fails to crack it. I suspect the problem is with gpg2john. To replicate

$ dd if=/dev/urandom bs=10000 count=1 > large
1+0 records in
1+0 records out
10000 bytes (10 kB) copied, 0.000673434 s, 14.8 MB/s
$ gpg -c --no-use-agent --passphrase 1234 --force-mdc large
$ ~/src/john/run/gpg2john large.gpg > key
$ ~/src/john/run/john  key
Warning: detected hash type "gpg", but the string is also recognized as "gpg-opencl"
Use the "--format=gpg-opencl" option to force loading these as that type instead
Using default input encoding: UTF-8
Loaded 1 password hash (gpg, OpenPGP / GnuPG Secret Key [32/64])
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status

and wait. There is no problem with mdc if the file is small;

$ dd if=/dev/urandom bs=1 count=1 > small
1+0 records in
1+0 records out
1 byte (1 B) copied, 4.6425e-05 s, 21.5 kB/s
$ gpg -c --no-use-agent --passphrase 1234 --force-mdc small
$ ~/src/john/run/gpg2john small.gpg > smallkey
$ ~/src/john/run/john  smallkey
Warning: detected hash type "gpg", but the string is also recognized as "gpg-opencl"
Use the "--format=gpg-opencl" option to force loading these as that type instead
Using default input encoding: UTF-8
Loaded 1 password hash (gpg, OpenPGP / GnuPG Secret Key [32/64])
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
1234             (?)
1g 0:00:00:00 DONE 2/3 (2016-04-04 20:50) 16.66g/s 8533p/s 8533c/s 8533C/s 123456..crawford
Use the "--show" option to display all of the cracked passwords reliably
Session completed

and also no problem if the file is large and mdc is off;

$ gpg -c --no-use-agent --passphrase 1234 --output large_no_mdc.gpg large 
$ ~/src/john/run/gpg2john large_no_mdc.gpg > key
[gpg2john] MDC is misssing, expect false positives!
$ ~/src/john/run/john  key
Warning: detected hash type "gpg", but the string is also recognized as "gpg-opencl"
Use the "--format=gpg-opencl" option to force loading these as that type instead
Using default input encoding: UTF-8
Loaded 1 password hash (gpg, OpenPGP / GnuPG Secret Key [32/64])
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
1234             (?)
1g 0:00:00:00 DONE 2/3 (2016-04-04 20:55) 11.11g/s 5688p/s 5688c/s 5688C/s 123456..crawford
Use the "--show" option to display all of the cracked passwords reliably
Session completed

[kholia]

We still have this problem even after commit 8f441513d25bf2b2fbee48c08af295646d0665e6. Simply bumping up the BIG_ENOUGH value in gpg_common.h does not fix this problem.

[kholia]

This bug seems to be in the handling of "partial" messages.

$ dd if=/dev/urandom bs=20000 count=1 > large
$ gpg -c --no-use-agent --passphrase 1234 --force-mdc large

$ pgpdump large.gpg 
Old: Symmetric-Key Encrypted Session Key Packet(tag 3)(13 bytes)
    New version(4)
    Sym alg - AES with 128-bit key(sym 7)
    Iterated and salted string-to-key(s2k 3):
        Hash alg - SHA1(hash 2)
        Salt - 66 e3 9f 8c cd dc e2 46 
        Count - 65536(coded count 96)
New: Symmetrically Encrypted and MDC Packet(tag 18)(8192 bytes) partial start
    Ver 1
    Encrypted data [sym alg is specified in sym-key encrypted session key]
        (plain text + MDC SHA1(20 bytes))
New:    (8192 bytes) partial continue
New:    (2048 bytes) partial continue
New:    (1024 bytes) partial continue
New:    (512 bytes) partial continue
New:    (130 bytes) partial end

gpg2john only processes the "partial start" 8192 bytes and skips over the remaining partial bytes. I don't see an easy way to fix this currently.

[jfoug]

I will see about this. This is the partial block, which is only 'valid' if we are not processing:

while (partial == YES) {
            // fprintf(stderr, "New: ");
            c = Getc();
            len = get_new_len(c);
            partial = is_partial(c);
            if (partial == YES)
                ;
                // fprintf(stderr, "\t(%d bytes) partial continue\n", len);
            else
                ;
                // fprintf(stderr, "\t(%d bytes) partial end\n", len);
            skip(len);
        }

If we are processing, we need to process.

We still have this problem even after commit 8f44151. Simply bumping up the BIG_ENOUGH value in gpg_common.h does not fix this problem.

This is because the 8192 size comes from the gpg blob. It appears to be 'chunked' (if the code is right). So increasing some static 'max' buffer size is not gonna cut it. If the file is chunked, then we have to process the chunks. Now, we 'could' allocate a 'big-enough' buffer and fill it. Big enough would be original size of file (binary files would be close, base-64 would easily be smaller. Then we could fill the buffer, and when we hit the end of the partial data, we have the 'real' full block. At least I think it should work that way.

[jfoug]

@kholia can you look at this. It does 'process' the extra data (at least for this type), but does not work. Likely we either need to read some header data OR not read the type byte. I will keep poking at it blindly, but if you had dox about how this data really is, then please take what I have started, and make it 'right'

none working diff code removed ......

[jfoug]

Got it! The above diff file has been removed. This one works (at least for this case. We will certainly need to make sure this works across the board, and clean things up a bit.

Edited one more time. Now both symmetric encryption handle long multi-part blocks.

diff --git a/src/gpg2john.c b/src/gpg2john.c
index a0aa273..5bbe849 100644
--- a/src/gpg2john.c
+++ b/src/gpg2john.c
@@ -120,6 +120,7 @@ private bz_stream bz;
 #include "jumbo.h"
 #include "misc.h"
 #include "params.h"
+#include "memory.h"
 #include "memdbg.h"    // Must be last included header

 #define YES 1
@@ -150,6 +151,7 @@ static int offset;
 static int gpg_dbg;
 static int dump_subkeys;
 static int is_subkey;
+static size_t m_flen;

 static int m_spec;
 static int m_algorithm;
@@ -227,13 +229,13 @@ public void multi_precision_integer(string);
 public void Reserved(int);
 public void Public_Key_Encrypted_Session_Key_Packet(int);
 public void Symmetric_Key_Encrypted_Session_Key_Packet(int);
-public void Symmetrically_Encrypted_Data_Packet(int);
+public void Symmetrically_Encrypted_Data_Packet(int,int,int);
 public void Marker_Packet(int);
 public void Literal_Data_Packet(int);
 public void Trust_Packet(int);
 public void User_ID_Packet(int);
 public void User_Attribute_Packet(int);
-public void Symmetrically_Encrypted_and_MDC_Packet(int);
+public void Symmetrically_Encrypted_and_MDC_Packet(int,int,int);
 public void Modification_Detection_Code_Packet(int);
 public void Private_Packet(int);

@@ -354,7 +356,12 @@ int gpg2john(int argc, char **argv)
    }

    for (i = 1; i < argc; ++i) {
+       FILE *fp;
        filename = argv[i];
+       fp = fopen(filename, "rb");
+       jtr_fseek64(fp, 0, SEEK_END);
+       m_flen = (size_t)jtr_ftell64(fp);
+       fclose(fp);
        if (freopen(filename, "rb", stdin) == NULL)
            warn_exit("can't open %s.", filename);
        parse_packet();
@@ -913,11 +920,23 @@ Symmetric_Key_Encrypted_Session_Key_Packet(int len)
 }

 public void
-Symmetrically_Encrypted_Data_Packet(int len)
+Symmetrically_Encrypted_Data_Packet(int len, int first, int partial)
 {
    int mode = get_sym_alg_mode();
-   char hash[2 * BIG_ENOUGH] = {0};
-   char *cp = hash;
+   static char *hash = NULL;
+   static char *cp;
+   static uint64_t totlen;
+
+   if (!hash)
+       hash = mem_alloc_tiny(m_flen+256, 2);
+   if (first) {
+       cp = hash;
+       totlen = 0;
+       // printf("\tVer %d\n", Getc());
+       Getc(); // version (we only read this from the first packet. Not read from rest of the 'partial' packets.
+   } else
+       ++len;  // we want the 'full' length for subsquent partial packets, since the logic is len-1 we simply fake it out.
+   totlen += (len-1);

    switch (mode) {
    case SYM_ALG_MODE_NOT_SPECIFIED:
@@ -940,24 +959,22 @@ Symmetrically_Encrypted_Data_Packet(int len)
    // The decrypted data will typically contain other packets (often
    // literal data packets or compressed data packets).

-   // m_usage is not really used in gpg_fmt_plug.c for symmetric hashes,
-   // let's hijack it for specifying tag values.
-   m_usage = 9; // Symmetrically Encrypted Data Packet (these lack MDC)
    give(len, m_data, sizeof(m_data));
-   if (len * 2 > BIG_ENOUGH - 128) {
-       fprintf(stderr, "[gpg2john] data is too large to be inlined, please file a bug!\n");
-   } else {
+   cp += print_hex(m_data, len - 1, cp);
+
+   if (!partial) {
+       // we only dump the packet out when we get the 'non-partial' packet (i.e. last one).
+
+       // m_usage is not really used in gpg_fmt_plug.c for symmetric hashes,
+       // let's hijack it for specifying tag values.
+       m_usage = 9; // Symmetrically Encrypted Data Packet (these lack MDC)
        fprintf(stderr, "[gpg2john] MDC is misssing, expect false positives!\n");
-       cp += sprintf(cp, "$gpg$*%d*%d*", m_algorithm, len); // m_algorithm == 0 for symmetric encryption?
-       cp += print_hex(m_data, len, cp);
-       cp += sprintf(cp, "*%d*%d*%d*%d", m_spec, m_usage, m_hashAlgorithm, m_cipherAlgorithm);
-       cp += sprintf(cp, "*%d*", m_count);
+       printf("$gpg$*%d*"LLd"*%s*%d*%d*%d*%d*%d*", m_algorithm, (long long)totlen, hash, m_spec, m_usage, m_hashAlgorithm, m_cipherAlgorithm, m_count);
+       cp = hash;
        cp += print_hex(m_salt, 8, cp);
-       puts(hash);
+       printf("%s\n", hash);
+       reset_sym_alg_mode();
    }
-
-   // skip(len);
-   reset_sym_alg_mode();
 }

 public void
@@ -1038,14 +1055,23 @@ User_Attribute_Packet(int len)
 }

 public void
-Symmetrically_Encrypted_and_MDC_Packet(int len)
+Symmetrically_Encrypted_and_MDC_Packet(int len, int first, int partial)
 {
    int mode = get_sym_alg_mode();
-   // printf("\tVer %d\n", Getc());
-   char hash[BIG_ENOUGH * 2] = {0};
-   char *cp = hash;
-
-   Getc(); // version
+   static char *hash = NULL;
+   static char *cp;
+   static uint64_t totlen;
+
+   if (!hash)
+       hash = mem_alloc_tiny(m_flen+256, 2);
+   if (first) {
+       cp = hash;
+       totlen = 0;
+       // printf("\tVer %d\n", Getc());
+       Getc(); // version (we only read this from the first packet. Not read from rest of the 'partial' packets.
+   } else
+       ++len;  // we want the 'full' length for subsquent partial packets, since the logic is len-1 we simply fake it out.
+   totlen += (len-1);
    switch (mode) {
    case SYM_ALG_MODE_SYM_ENC:
        // printf("\tEncrypted data [sym alg is specified in sym-key encrypted session key]\n");
@@ -1056,24 +1082,17 @@ Symmetrically_Encrypted_and_MDC_Packet(int len)
        break;
    }
    give(len - 1, m_data, sizeof(m_data));
-   if (len * 2 > BIG_ENOUGH - 128) {
-       fprintf(stderr, "[gpg2john] data is too large to be inlined, please file a bug!\n");
-   } else {
-       cp += sprintf(cp, "$gpg$*%d*%d*", m_algorithm, len - 1); // m_algorithm == 0 for symmetric encryption?
-       cp += print_hex(m_data, len - 1, cp);
+   cp += print_hex(m_data, len - 1, cp);
+
+   if (!partial) {
+       // we only dump the packet out when we get the 'non-partial' packet (i.e. last one).
        m_usage = 18; // Sym. Encrypted Integrity Protected Data Packet (Tag 18)
-       cp += sprintf(cp, "*%d*%d*%d*%d", m_spec, m_usage, m_hashAlgorithm, m_cipherAlgorithm);
-       cp += sprintf(cp, "*%d*", m_count);
+       printf("$gpg$*%d*"LLd"*%s*%d*%d*%d*%d*%d*", m_algorithm, (long long)totlen, hash, m_spec, m_usage, m_hashAlgorithm, m_cipherAlgorithm, m_count);
+       cp = hash;
        cp += print_hex(m_salt, 8, cp);
-       if (m_usage == 1) { /* handle 2 byte checksum */
-           fprintf(stderr, "Symmetrically_Encrypted_and_MDC_Packet doesn't handle 2 bytes checksums yet!\n");
-       }
-       puts(hash);
+       printf("%s\n", hash);
+       reset_sym_alg_mode();
    }
-
-   // printf("\t\t(plain text + MDC SHA1(20 bytes))\n");
-   // skip(len - 1); // we did "give()" already
-   reset_sym_alg_mode();
 }

 /* this function is not used because this packet appears only
@@ -1448,7 +1467,7 @@ is_partial(int c)
 }

 public void
-parse_packet(void)
+parse_packet()
 {
    int c, tag, len = 0;
    int partial = NO;
@@ -1527,11 +1546,11 @@ parse_packet(void)

        if (tag < TAG_NUM && tag_func[tag] != NULL) {
            if (gpg_dbg)
-               fprintf(stderr, "Packet type %d, len %d at offset %d  (Processing) (pkt-type %s)\n", tag, len, offset, pkt_type(tag));
-           (*tag_func[tag])(len);
+               fprintf(stderr, "Packet type %d, len %d at offset %d  (Processing) (pkt-type %s) (Partial %s)\n", tag, len, offset, pkt_type(tag), partial?"yes":"no");
+           (*tag_func[tag])(len, 1, partial);  // first packet (possibly only one if partial is false).
        } else {
            if (gpg_dbg)
-               fprintf(stderr, "Packet type %d, len %d at offset %d  (Skipping)\n", tag, len, offset);
+               fprintf(stderr, "Packet type %d, len %d at offset %d  (Skipping) (Partial %s)\n", tag, len, offset, partial?"yes":"no");
            skip(len);
        }
        while (partial == YES) {
@@ -1539,13 +1558,20 @@ parse_packet(void)
            c = Getc();
            len = get_new_len(c);
            partial = is_partial(c);
-           if (partial == YES)
-               ;
-               // fprintf(stderr, "\t(%d bytes) partial continue\n", len);
-           else
-               ;
-               // fprintf(stderr, "\t(%d bytes) partial end\n", len);
-           skip(len);
+           if (partial == YES) {
+               if (gpg_dbg)
+                   fprintf(stderr, "\t(%d bytes) partial continue\n", len);
+           }
+           else {
+               if (gpg_dbg)
+                   fprintf(stderr, "\t(%d bytes) partial end\n", len);
+           }
+           if (tag < TAG_NUM && tag_func[tag] != NULL) {
+               if (gpg_dbg)
+                   fprintf(stderr, "Packet type %d, len %d at offset %d  (Processing) (pkt-type %s) (Partial %s)\n", tag, len, offset, pkt_type(tag), partial?"yes":"no");
+               (*tag_func[tag])(len, 0, partial);  // subsquent packets.
+           } else
+               skip(len);
        }
        if (len == EOF) return;
    }

[jfoug]

I will create a PR for this, once the branch builds on the CI's properly. 01b35b0

[jfoug]

PR made #2271 and here is run from 'original' post on this thread

$ dd if=/dev/urandom bs=10000 count=1 > large
1+0 records in
1+0 records out
10000 bytes (10 kB, 9.8 KiB) copied, 0.00311157 s, 3.2 MB/s
$ gpg -c --no-use-agent --passphrase 1234 --force-mdc large
$ ../run/gpg2john large.gpg > key
$ ../run/john key
Using default input encoding: UTF-8
Loaded 1 password hash (gpg, OpenPGP / GnuPG Secret Key [32/64])
Will run 2 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
1234             (?)
1g 0:00:00:00 DONE 2/3 (2016-09-12 23:13) 16.66g/s 2133p/s 2133c/s 2133C/s 123456..john
Use the "--show" option to display all of the cracked passwords reliably
Session completed

But I would like someone else to look the code over. NOTE, I did not test the non-MDC function. It is written, and 'should' work, but it was not tested

[jfoug]

After th at merge, I see a problem. I allocate buffer 'hash' 1 time, if it is null. But if we are processing multiple files, that will simply make it large enough to hold the first file! It needs to be allocated each file (large enough), and then passed into the processing function.

[jfoug]

I have the gpg2john working properly now (and will check in). Moving on to the next problem. gpg_fmt needs to be re-done to remove 'BIG_ENOUGH' totally.

[jfoug]

PR #2272 (once tested and approved), should 'fix' the problems in cracking large gpg encrypted blobs .

[jfoug]

7147798 ASAN problems (note, these were likely there before I started).

[jfoug]

with PR #2272 merged, this issue is now fixed.