BFEgg: -fork sometimes puts multiple entries into .pot file, that are different.

[jfoug]

Does not happen often, and I am not sure it happens for all OS's.

I have seen this on cygwin.

Here is an example:

form=BFegg                        guesses: 1500 0:00:00:00 DONE  [PASSED]
FAILED line = 1                (11)
.pot CHK:BFegg                    guesses: 1501 -show=1500 0:00:00:00 DONE : Expected count(s) (1500)  [!!!FAILED4!!!]  (1500 val-pwd  1 inval-pwd)
Exiting on error. The .pot file ./tst.pot contains the found data
The command used to run this test was:

After some digging, I found this:

$ grep -a V6ZOx0rVGWT0 tst.pot
+V6ZOx0rVGWT0:11
+V6ZOx0rVGWT0:1

So I have +V6ZOx0rVGWT0 input being found (as seen by entries in the john.pot file) for 1 and 11

The real value is 1.

Note, I am only seeing this on cygwin for sure, and only on -fork code. This was run under the TS, but it should be pure john, since all the TS is doing is running john. john creates the .pot file totally on it's own.

[magnumripper]

Have you seen other formats than BFegg do this?

[magnumripper]

This seems to be a BFegg-specific problem. Also, TS is not very clever here:

$ ./jtrts.pl -pass=-fork=4 -stop bfegg
-------------------------------------------------------------------------------
- JtR-TestSuite (jtrts). Version 1.13, Dec 21, 2014.  By, Jim Fougeron & others
- Testing:  John the Ripper password cracker, version 1.8.0.2-jumbo-1-bleeding_omp [darwin14.1.0 64-bit AVX-autoconf]
--------------------------------------------------------------------------------

John Jumbo build detected.

form=BFegg                        guesses: 1500 0:00:00:00 DONE  [PASSED]
.pot CHK:BFegg                    guesses: 1501 0:00:00:00 DONE  [PASSED] (1501 val-pwd)

All tests passed without error.  Performed 1 tests.  Time used was 2 seconds
$ cut -d: -f1 tst.pot|sort|uniq -d
cut: tst.pot: No such file or directory

Why did the .pot check consider 1501 to be an OK figure after cracking 1500? Since it did, -stoponerror did not kick in so I have no tst.pot file to examine.

[magnumripper]

$ ./jtrts.pl -pass=-fork=4 -stop bfegg
-------------------------------------------------------------------------------
- JtR-TestSuite (jtrts). Version 1.13, Dec 21, 2014.  By, Jim Fougeron & others
- Testing:  John the Ripper password cracker, version 1.8.0.2-jumbo-1-bleeding_omp [darwin14.1.0 64-bit AVX-autoconf]
--------------------------------------------------------------------------------

John Jumbo build detected.

form=BFegg                        guesses: 1500 0:00:00:00 DONE  [PASSED]
.pot CHK:BFegg                    guesses: 1502 -show=1500 0:00:00:00 DONE : Expected count(s) (1500)  [!!!FAILED!!!]  (1501 val-pwd)
Exiting on error. The .pot file ./tst.pot contains the found data
The command used to run this test was:

../run/john -ses=./tst  -fork=4 -pot=./tst.pot BFegg_tst.in --wordlist=pw.dic -form=bfegg

$ LC_ALL=C cut -d: -f1 tst.pot|sort|uniq -d
+.QN6k.Zodje.
+V6ZOx0rVGWT0

$ grep V6ZOx0rVGWT0 tst.pot 
+V6ZOx0rVGWT0:1
+V6ZOx0rVGWT0:11

$ grep .QN6k.Zodje. tst.pot 
+.QN6k.Zodje.:!!!!!!!!!
+.QN6k.Zodje.:!!

Probably a buffer cleaning issue.

[magnumripper]

I was sure this would fix it, but it didn't

diff --git a/src/BFEgg_fmt_plug.c b/src/BFEgg_fmt_plug.c
index fd027d0..0065cbf 100644
--- a/src/BFEgg_fmt_plug.c
+++ b/src/BFEgg_fmt_plug.c
@@ -189,6 +189,8 @@ static int crypt_all(int *pcount, struct db_salt *salt)
                if (saved_key[index][0] != 0)
                        blowfish_encrypt_pass(saved_key[index],
                                (char*)crypt_out[index]);
+               else
+                       memset(crypt_out[index], 0, BINARY_SIZE);
        }
        return count;
 }

[jfoug]

Only affects fork

[magnumripper]

No, it's just that fork make it surface (like the DES 8-bit issue). They are actually valid candidates. This should be FMT_NOT_EXACT.

Try this, still passes self-test

diff --git a/src/BFEgg_fmt_plug.c b/src/BFEgg_fmt_plug.c
index fd027d0..614b419 100644
--- a/src/BFEgg_fmt_plug.c
+++ b/src/BFEgg_fmt_plug.c
@@ -60,6 +60,8 @@ static struct fmt_tests tests[] = {
     {"+EEHgy/MBLDd0", "walkman"},
     {"+vPBrs07OTXE/", "tesztuser"},
     {"+zIvO/1nDsd9.", "654321"},
+    {"+V6ZOx0rVGWT0", "1"},
+    {"+V6ZOx0rVGWT0", "11"},
     {NULL}
 };

[magnumripper]

You can trigger it with -random instead of -fork but you don't see it (eg. that it's sometime "1" and sometimes "11"). If you set FMT_NOT_EXACT you will also see it.

[frank-dittrich]

Is the BFegg format "not exact", or is our implementation broken?

[magnumripper]

Not sure. Does pass_gen.pl produce the same for 1 and 11?

[magnumripper]

Yep... and worse.

$ ../run/pass_gen.pl bfegg

Enter words to hash, one per line.
When all entered ^D starts the processing.

  ** Here are the hashes for format  **
1
u0:+V6ZOx0rVGWT0:0:0:1:
11
u1:+V6ZOx0rVGWT0:1:0:11:
111
u2:+V6ZOx0rVGWT0:2:0:111:
1111
u3:+V6ZOx0rVGWT0:3:0:1111:
11111
u4:+V6ZOx0rVGWT0:4:0:11111:
111111
u5:+V6ZOx0rVGWT0:5:0:111111:
1111111
u6:+V6ZOx0rVGWT0:6:0:1111111:

[magnumripper]

Our format seem to be totally b0rken and unfortunately the TS (via pass_gen.pl) appears to implementing the same bugs (which defeats the purpose with TS). Here's probably correct hashes:

$ perl -ne 'use Authen::Passphrase::EggdropBlowfish; $ppr = Authen::Passphrase::EggdropBlowfish->new(passphrase => $_); print $ppr->hash_base64, "\n";'
1
OjwL20zG.Ty1
11
Ua4KX/ZWGEv0
111
/ZdoY/FvKKk1
1111
4zK2n0QEfK50

[magnumripper]

Scrap my last statement. I missed a chomp. Our code is correct, the algo is b0rken.

$ perl -ne 'use Authen::Passphrase::EggdropBlowfish; chomp; $ppr = Authen::Passphrase::EggdropBlowfish->new(passphrase => $_); print $ppr->hash_base64, "\n";'
1
V6ZOx0rVGWT0
11
V6ZOx0rVGWT0
111
V6ZOx0rVGWT0
1111
V6ZOx0rVGWT0
11111
V6ZOx0rVGWT0
111111
V6ZOx0rVGWT0

[magnumripper]

I ran rockyou (len 1-72) through BFegg. That's 14,343,560 hashes but only 14,309,085 unique ones. This format has a 56-bit binary, so should be better than that.

[magnumripper]

I committed adding FMT_NOT_EXACT but maybe we don't want that? If we revert it, a user could use the --keep-guessing option if she really wants to see alternatives.

[magnumripper]

Top-10 collisions from Rockyou:

  58 V6ZOx0rVGWT0
  57 v.gq8.qm3rM1
  50 AjpZu1fTJ6./
  39 KP4D8.s.zp6/
  34 YbuKH0cOnBj/
  34 YFmov.KZTp01
  33 ssvKP1EXwvj0
  32 e0JBR.zRmIG/
  29 GEJRM/RmokD/
  28 qZ6fN1HIXS31
  28 5Idqs19PtTc0
  27 pzBc0064GIE1

58 different words was hashed to V6ZOx0rVGWT0, and so on.

[magnumripper]

It seems any repetition of a pattern (which can be longer than 1 character) always gets same hash as a single instance of it.

$ perl -ne 'use Authen::Passphrase::EggdropBlowfish; chomp; if (length() && length() <= 72) { $ppr = Authen::Passphrase::EggdropBlowfish->new(passphrase => $_); print $ppr->hash_base64, "\n";}' 
x
3Fi9q.glmnd0
xx
3Fi9q.glmnd0
hello
4TLS6/5.KHe1
hellohello
4TLS6/5.KHe1

I can't find any mentioning of this on the 'net.

[jfoug]

I am not sure what to do on the TS. I guess it simply gets left to show the error. The TS was not designed for having multiple "right" guesses any of which are as valid as any other.

[magnumripper]

One way to fix it is using another (or a shorter) dictionary that does not include any repeated patterns. Maybe your existing logic for shortening the input dictionary (for slow formats)?